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We study the Fermi surface instabilities of the Pomeranchuk type in the spin triplet channel with 
high orbital partial waves (F° (I > 0)). The ordered phases are classified into two classes, dubbed the 
a and /3-phases by analogy to the superfluid 3 He-A and B-phases. The Fermi surfaces in the a-phases 
exhibit spontaneous anisotropic distortions, while those in the /3-phases remain circular or spherical 
with topologically non-trivial spin configurations in momentum space. In the a-phase, the Goldstone 
modes in the density channel exhibit anisotropic overdamping. The Goldstone modes in the spin 
channel have nearly isotropic underdamped dispersion relation at small propagating wavevectors. 
Due to the coupling to the Goldstone modes, the spin wave spectrum develops resonance peaks in 
both the a and /3-phases, which can be detected in inelastic neutron scattering experiments. In 
the p-wave channel /3-phase, a chiral ground state inhomogeneity is spontaneously generated due to 
a Lifshitz-like instability in the originally nonchiral systems. Possible experiments to detect these 
phases are discussed. 

PACS numbers: 71.10.Ay, 71.10.Ca, 05.30.Fk 



I. INTRODUCTION 

The Landau theory of the Fermi liquid is one of the 
most successful theories of condensed matter physics 
2]. It describes a stable phase of dense interacting 
fcrmionic systems, a Fermi liquid (FL). Fermi liquid the- 
ory is the foundation of our understanding of conven- 
tional, weakly correlated, metallic systems. Its central as- 
sumption is the existence of well-defined fermionic quasi- 
particles, single particle fermionic excitations which exist 
as long-lived states at very low energies, close enough to 
the Fermi surface. In the Landau theory, the interac- 
tions among quasi-particles are captured by a few Lan- 
dau parameters F^ 0, , where I denotes the orbital angular 
momentum partial wave channel, and s, a denote spin 
singlet and triplet channels, respectively. Physical quan- 
tities, such as the spin susceptibility, and properties of 
collective excitations, such as the dispersion relation of 
zero sound collective modes, acquire significant but finite 
renormalizations due to the Landau interactions. In the 
FL phase, except for these finite renormalizations, the 
effects of the interactions become negligible at asymp- 
totically low energies. It has, however, long been known 
that the stability of the FL requires that the Landau pa- 
rameters cannot be too negative, Ff' a > — (21 + 1), a 
result first derived by Pomeranchuk Q . The most famil- 
iar of these Pomeranchuk instabilities are found in the 
s-wave channel: the Stoner ferromagnetism at Fft < — 1 
and phase separation at Fq < — 1. 

It has been realized quite recently that when these 
bounds are violated in a channel with a non-vanishing 
angular momentum, there is a ground state instability 
in the particle-hole, spin singlet, channel leading to a 
spontaneous distortion of the Fermi surface. This is a 
quantum phase transition to a uniform but anisotropic 
liquid phase of the fermionic system Q ■ In such a phase 



the electron fluid behaves, from the point of view of its 
symmetries and of their breaking, very much like an elec- 
tronic analog of liquid crystal phases [1, 0| . The charge 
nematic is the simplest example of such electronic liquid 
crystal phases, a concept introduced in Ref. [7| to de- 
scribe the complex phases of strongly correlated systems 
such as doped Mott insulators. The charge nematic phase 
has also been suggested to exist in the high T c materi- 
als near the melting of the (smectic) stripe phases 0, H| , 
and in quantum Hall systems in nearly half-filled Landau 
levels [t| E3] • Experimentally, the charge nematic phase 
has been found in ultra-high mobility two-dimensional 
electron gases (2DEG) in AlAs-GaAs heterostructures 
and quantum wells in large magnetic fields, in nearly 
half filled Landau levels for N > 2 at very low tem- 
peratures [l(| EH, EH EH • Strong evidence for a charge 
nematic phase has been found quite recently near the 
metamagnetic transition of the ultra-clean samples of the 
bilayer ruthenate St^R^Oi [H], EH- 

Electronic liquid crystal phases can also be realized 
as Pomeranchuk instabilities in the particle-hole channel 
with non-zero angular momentum. This point of view 
has been the focus of much recent work, both in con- 
tinuum system [J, \pl fijj, El) EH as well as in lattice 
systems (20. [2ll \22l |23|. |24| ■ in which case these quantum 
phase transition involve the spontaneous breaking of the 
point symmetry group of the underlying lattice. The 2D 
quantum nematic Fermi fluid phase is an instability in the 
d-wave (I = 2) channel exhibiting a spontaneous ellipti- 
cal distortion of the Fermi surface [3] • In all cases, these 
instabilities typically result in anisotropic Fermi surface 
distortions, sometimes with a change of the topology of 
the Fermi surface. Nematic phases also occur in strong 
coupling regimes of strongly correlated systems such as 
the Emery model of the high temperature superconduc- 
tors [25[ . If lattice effects are ignored, the nematic state 
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has Goldstone modes can be viewed as the rotation of the 
distorted Fermi surface, i.e., the soft Fermi surface fluctu- 
ations. Within a random phase approximation (RPA) ap- 
proach Q, confirmed by a non-pert urbative high dimen- 
sional bosonization treatment [ljj , the Goldstone modes 
were shown to be overdamped in almost all the propagat- 
ing directions, except along the high symmetry axes of 
the distorted Fermi surface. The Goldstone mode couples 
strongly to electrons, giving rise to a non- Fermi liquid be- 
havior throughout the nematic Fermi fluid phase [1, Hi| : 
in perturbation theory, the imaginary part of the elec- 
tron self-energy is found to be proportional to ws on 
most of the Fermi surface, except along four "nodal" di- 
rections, leading to the breakdown of the quasi-particle 
picture. Away from the quantum critical point, this ef- 
fect is suppressed by lattice effects but it is recovered at 
quantum criticality [2l[ and at high temperatures (if the 
lattice pinning effects are weak). Beyond perturbation 
theory [26( this effect leads to a form of 'local quantum 
criticality'. 

Richer behaviors still can be found in Pomeranchuk 
instabilities in the s pin triplet and I > angular momen- 
tum channels 0, S, M, 0, M, H ED, M, \m El ■ Typ- 
ically, these class of instabilities break both the spacial 
(orbital) and spin SU (2) rotation symmetries. A p-wave 
channel instability was studied by Hirsch [2?], [28| ■ The 
Fermi surfaces of spin up and down components in such 
a state shift along opposite directions. The p-wave chan- 
nel instability was also proposed by Varma as a candidate 
for the hidden order app earing in the heavy fermion com- 
pound URu2Si2 [HI, [32] below 17 K. Gor'kov and Sokol 
studied the non-Neel spin orderings in itinerant systems, 
and showed their relation to the Pomeranchuk instabil- 
ity [29[ . Oganesyan and coworkers 0, [1] proposed the ex- 
istence of a nematic-spin-nematic phase, a nematic state 
in both real space and in the internal spin space, where 
the two Fermi surfaces of up and down spins are sponta- 
neously distorted into two orthogonal ellipses. Podolsky 
and Demler 35] considered a spin-nematic phase as aris- 
ing from the melting of a stripe phase; this phase can 
also be a nematic-spin-nematic if it retained a broken 
rotational symmetry, as expected from the melting of a 
stripe (or smectic) phase @, H^. Kee and Kim [33| have 
suggested a state to explain the behavior of St^R^O-; 
which can be shown to be a lattice version of a partially 
polarized nematic-spin-nematic state. 

Interestingly, the spin triplet Pomeranchuk instabili- 
ties can also occur without breaking rotational invariance 
in real space, and keep the symmetries of the undistorted 
Fermi surface. Wu et al. [30] showed that a state of 
this sort can exist in the p-wave channel, with the dis- 
tortions affecting only the spin channel. In this state, 
there are two Fermi surfaces with different volumes, as 
in ferromagnetic systems. Although both the spatial and 
spin rotation symmetries are broken, a combined spin- 
orbit rotation keeps the system invariant, i.e., the total 
angular momentum is conserved. The broken symme- 
try in such a state is the relative spin-orbit symmetry 



which was first proposed by Leggett in the context of 
superfluid 3 He-B phase In fact, it is a particle-hole 
channel analog of the superfluid 3 He-B phase where the 
pairing gap function is isotropic over the Fermi surface. 
The phases with anisotropic Fermi surfaces distortions 
are the analog of the 3 He- A phase where the gap func- 
tion is anisotropic. The two possibilities of keeping or 
breaking the shape of Fermi surfaces are dubbed (3 and 
a-phases respectively, by analogy with B and A super- 
fluid phases in 3 He systems. An important difference is, 
however, that while in the A and B phases of superfluid 
3 He all the Fermi surface is gapped, up to a set of mea- 
sure zero of nodal points in the A phase, in the (1 and 
a-phases of spin triplet Pomeranchuk systems no gap in 
the fermionic spectrum ever develops. 

An important common feature of the a and /3-phases 
is the dynamical appearance of effective spin-orbit (SO) 
couplings, reflecting the fact that in these phases spin and 
orbit degrees of freedom become entangled [37j • Con- 
ventionally, in atomic physics, the SO coupling origi- 
nates from leading order relativistic corrections to the 
Schrodinger-Pauli equation. As such, the standard SO 
effects in many-body systems have an inherently single- 
particle origin, and are unrelated to many-body correla- 
tion effects. The Pomeranchuk instabilities involving spin 
we are discussing here thus provide a new mechanism 
to generate effective SO couplings through phase tran- 
sitions in a many-body non-relativistic systems. In the 
2D /3-phase, both Rashba and Dresselhaus-like SO cou- 
plings can be generated. In the a-phase, the resulting SO 
coupling can be considered as a mixture of Rashba and 
Dresselhaus with equal coupling strength. Such SO cou- 
pling systems could in principle be realized in 2D semi- 
conductor materials leading to interesting new effects. 
For instance, a hidden SU (2) symmetry in such systems 
was found to give rise to a long lived spin spiral excita- 
tion with the characteristic wavevector relating the two 
Fermi surfaces together [38]. Recently, many proposals 
have been suggested to employ SO coupling in semicon- 
ductor materials to generate spin current through electric 
fields. The theoretical prediction of this "intrinsic spin 
Hall effect" [1^, H3] has stimulated tremendous research 
activity both theoretical and experimental [Il|,[42]. Thus, 
Pomeranchuk instabilities in the spin channel may have 
a potential application to the field of spintronics. 

A systematic description of the high partial- wave chan- 
nel Pomeranchuk instabilities involving spin is still lack- 
ing in the literature. In this paper, we investigate this 
problem for arbitrary orbital partial wave channels in two 
dimensions (2D), and for simplicity only in the p-wave 
channel in three dimensions (3D). We use a microscopic 
model to construct a general Ginzburg-Landau (GL) free 
energy to describe these instabilities showing that the 
structure of the a and /3-phases are general for arbitrary 
values of I . The a-phases exhibit anisotropic relative dis- 
tortions for two Fermi surfaces as presented in previous 
publications. We also investigate the allowed topological 
excitations (textures) of these phases and find that a half- 



3 



quantum vortex-like defect in real space, combined with 
spin-orbit distortions. The /3-phases at I > 2 also have 
a vortex configuration in momentum space with wind- 
ing numbers ±Z which are equivalent to each other by a 
symmetry transformation. 

We study the collective modes in critical regime and 
in the ordered phases at zero temperature. At the quan- 
tum critical point, as in the cases previously studied, the 
theory has dynamic critical exponent z = 3 for all values 
of the orbital angular momentum I. In the anisotropic 
a-phase, the Goldstone modes can be classified into den- 
sity and spin channel modes, respectively. The density 
channel Goldstone mode exhibits anisotropic overdamp- 
ing in almost all the propagating directions. In contrast, 
the spin channel Goldstone modes show nearly isotropic 
underdamped dispersion relation at small propagating 
wavevectors. In the /3-phase, the Goldstone modes are 
relative spin-orbit rotations which have linear dispersion 
relations at I > 2, in contrast to the quadratic spin- wave 
dispersion in the ferromagnet. Both the Goldstone modes 
(spin channel) in the a-phases and the relative spin-orbit 
Goldstone modes in the /3-phases couple to spin excita- 
tions in the ordered phases. Thus the spin wave spec- 
tra develop characteristic resonance peaks observable in 
neutron scattering experiments, which are absent in the 
normal phase. 

The p-wave channel Pomeranchuk instability involving 
spin is special because the Ginzburg-Landau (GL) free 
energy contains a cubic term of order parameters with a 
linear spatial derivative satisfying all the symmetry re- 
quirement. Such a term is not allowed in other chan- 
nels with I 1, including the ferromagnetic instability 
and the Pomeranchuk instabilities in the density chan- 
nel. This term does not play an important role in the ex- 
phase. But in the /5-phase, it induces a chiral inhomoge- 
neous ground state configuration leading to a Lifshitz-likc 
instability in this originally nonchiral system. In other 
words, a Dzyaloshinskii-Moriya type interaction for the 
Goldstone modes is generated in the /3-phase. The effect 
bears some similarity to the helimagnet [43| and chiral 
liquid crystal @ except that the parity is explicitly bro- 
ken there but not here. The spiral pattern of the ground 
state order parameter is determined. 

The paper is organized as follows: In Section HH we 
construct the model Hamiltonian for the Pomeranchuk 
instabilities with spin. In Section IIII1 we present the 
Ginzburg-Landau free energy analysis to determine the 
allowed ground states. In Section IIV[ we discuss the re- 
sults of the mean field theory. In Section [V] we discuss 
the topology of the broken symmetry states and classify 
the topological defects. In Section IVI1 we calculate the 
collective modes at the critical point at zero tempera- 
ture. In Section [VTU and Section IVlTTl we investigate 
the Goldstone modes in the a and /3-phases respectively, 
and study the spontaneous Lifshitz instability in the 0- 
phase at I = 1. In Section ILXl we present the magnetic 
field effects. In Section [Xj we show that in the a and 
/3-phases in the quadrupolar channel (1 = 2), a spin cur- 



rent can be induced when a charge current flows through 
the system. In Section [Xll we discuss the possible exper- 
imental evidence for Pomeranchuk instabilities involving 
spins. We summarize the results of this paper at Section 
IXIII Details of our calculations are presented in two ap- 
pendices. In Appendix IA1 we specify our conventions for 
Landau parameters and in Appendix [B] we give details of 
the Goldstone modes for spin oscillations in the a-phase. 

II. MODEL LANDAU HAMILTONIAN 

We begin with the model Hamiltonian describing the 
Pomeranchuk instability in the F ; a (I > 1) channel in 2D. 
Later on in the paper we will adapt this scheme to discuss 
the 3D case which is more complex. This model, and 
the related model for the spin-singlet sector of Ref. Q 
on which it is inspired, has the same structure as the 
effective Hamiltonian for the Landau theory of a FL. The 
corresponding order parameters can be defined through 
the matrix form as 

Q» b (r) = ^ a (r)a^ 9l , b (-iV)Mr) (2-1) 

where the Greek indices /i denote the x, y, z directions 
in the spin space, and in 2D the latin indices b = 1,2 
denote the two orbital components, and a,/3 =|, J, label 
the two spin projections. (Hereafter, repeated indices are 
summed over.) In 2D, the operators gi^i ± igi,2, which 
carry the azimuthal angular momentum quantum num- 
ber L z = ±Z, are given by 

<? U HV) ± igiA-W) = H)'(Vs ± Wy)\ (2.2) 

where the operator V a is defined as V a /|V|. The 3D 
counterpart of these expressions can be written in terms 
of spherical harmonic functions. Thus, in 3D the latin 
labels take 21 + 1 values. For the moment, and for sim- 
plicity, we will discuss first the 2D case. 

In momentum space (i.e. a Fourier transform) we can 
write the operators of Eq. (|2.2p in the form gi.i(k) = 
cosWk and gi t i(k) = sinWk, where Ok is the azimuthal 
angle of k in the 2D plane. In momentum space, Q^ lb (q) 
is defined as 

Q" b (q) = + f) <f> Mk- -)• (2.3) 

It satisfies Q^i-q) = Q^ A *(q), thus Q^ b {r) is real. 

We generalize the Hamiltonian studied in Ref. [!, [3(| 
to the Ff 1 channel with arbitrary values of I as 

H = j d 2 r Vi(r)(e(V) - ^„(f) 

+ \ J d d rdV / ; a (r-r')^Q^(r)(5^(r'), 

fib 

(2.4) 



4 



where /j, is the chemical potential. For later convenience, 
we include the non-linear momentum dependence in the 
single particle spectrum up to the cubic level as 

e(fc) = v F Ak(l + a{Ak/k F ) + b{Ak/k F ) 2 + . . .) (2.5) 

with Afc = k — kp. Here v F and k F are the Fermi velocity 
and the magnitude of the Fermi wave vector in the FL. 

The Fourier transform of the Landau interaction func- 
tion f(r) is 



f(q) = / dfe^ft(r) = 



f? 



(2.6) 



i + *l/f|g 2 ' 

and the dimensionless Landau parameters are defined as 



Ft = N(0)ft(q = 0) 



(2.7) 



with N(0) the density of states at the Fermi energy. This 
Hamiltonian possesses the symmetry of the direct prod- 
uct of SOl(2) &> SOs(3) in the orbit and spin channels. 

The LP instability occurs at F? < — 2 at I > 1 in two 
dimensions. For the general values of I, we represent the 
order parameter by a 3 x 2 matrix 



\m 



d 2 k 



j^Wl{kK p 9i,b(k)Mk))- (2-8) 



It is more convenient to represent each column of the 
matrix form n fi ' b (b = 1, 2) as a 3- vector in spin space as 



ni =n"' 1 , 



n 2 = n^ 2 . 



(2.9) 



For / = 1, fix, 2 are just the spin currents along the x,y 
directions respectively. When I > 2, Hi t 2 denote the spin 
multipole components at the level / on the Fermi surface, 
ni ± in-2 carry the orbital angular momenta L z = ±1 
respectively. In other words, Hi >2 are the counterpart 
of the spin-moment in the l-th partial wave channel in 
momentum space. 

The mean field Hamiltonian, i.e. for a state with a 
uniform order parameter, can be decoupled as 



H 



MF 



d 2 k 



n 2 sm(W k )j ■ <x}^g(fc) 



(ni cos(Wk) 



K| 2 + K| 2 



(2.10) 



This mean field theory is valid when the interaction range 
£ ~ V K l/; a | i s much larger than the inter-particle dis- 
tance d w \/k F . The actual validity of mean field theory 
at quantum criticality requires an analysis of the effects 
of quantum fluctuations which are not included in mean 
field theory [H, |45[ . In this theory, just as the case of 
Ref. [1], the dynamic critical exponent turns out to be 
z = 3 and mean field theory appears to hold even at 
quantum criticality. 



Taking into account that |/"| ~ 1/N(0) around the 
transition point, we introduce a dimensionless parameter 
A to denote the above criterion as 



A = 



N(0) 



» 1. 



(2.11) 



Finally, notice that, in the p-wave channel, the Hamilto- 
nian Eq. (|2.10p can be formally represented through an 
SU(2) non-Abelian gauge field minimally-coupled to the 
fermions 



mf 



d 2 r —ipi(r)(-it£V a - mA^{f)a^) 2 ^{r) 



2m 

where the gauge field is defined as 

i4£(tV =n"°(TV. 



(2.12) 



(2.13) 



Notice, however, that this is an approximate effective lo- 
cal gauge invariance which only holds for a theory with a 
linear dispersion relation, and that it is manifestly bro- 
ken by non-linear corrections, such as the quadratic term 
of Eq. (|2.12|) . and the cubic terms included in the dis- 
persion e(k) of Eq. (|2.5[) . 



III. GINZBURG-LANDAU FREE ENERGY 

A. The 2D systems 

In order to analyze the possible ground state configu- 
ration discussed in Section |H we construct the G-L free 
energy in 2D in the arbitrary Z-wave channel. The sym- 
metry constraint to the G-L free energy is as follows. Un- 
der time-reversal (TR) and parity transformations, n\ t 2 
transform, respectively, as 



Tni j2 T 



(-) i+i ni,2, Pni, 2 P- L = R%, 2 . (3.1) 



Under the 5*0^(3) rotation in the spin channel, 



"1,2 



transform as 



(3.2) 



On the other hand, under a uniform rotation by an an- 
gle about the z-axis, in the orbital channel, the order 
parameters n a transform as 



Hi — > cos(W) Hi + sm(W) H 2 , 
H2 — > — shx{lff) Hi + cos(W) n 2 . 



(3.3) 



Thus, the order parameter fields Hi and n 2 are invariant 
under spatial rotations by 2w/l, and change sign under 
a rotation by tt/1. In the a-phase this change change be 
compensated by flipping the spins. 



5 



In order to maintain the SOl (2) ® ^Og^) symmetry, 
up to quartic terms in the order parameter fields n a , the 
uniform part of the GL free energy has the form 



F(n) = rtr(n T n) + («! + y)[tr(n T n)] 2 - ytr[(?i T n) 2 



| a + |r*-' 2 
+ u 2 ]ni x ri 2 



r(|ni| 2 + |n 2 | 2 )+wi(|ni| 2 + |n 2 

2 



2\2 



(3.4) 



The coefficients r, v\ , v 2 will be presented in Eq. (|4.12|) 
by evaluating the ground state energy of the mean field 
Hamiltonians Eq. (|4.1[) and Eq. (|4.7[) . The Pomeranchuk 
instability occurs at r < 0, i.e., < — 2 (/ > 1). Fur- 
thermore, for v 2 > 0, the ground state is the a-phase 
which favors ri\ || ri 2 , while leaving the ratio of |ni|/|n 2 | 
arbitrary. On the other hand, for v 2 < we find a /3- 
phase, which favors fti _L ri 2 and |ni| = |n 2 |. 

The gradient terms are more subtle. We present the 
gradient terms of the GL free energy for I = 1 as follows 

F g rad(n) = -fitiid a n T d a n}+ l2 €^ x n^ a n" b d a n xb 
= li{d a n b ■ d a ri b ) + ~/ 2 [(d x ri 2 - d y n x ) 

■ (nixn 2 )}. (3.5) 

For simplicity, as in Ref. we have neglected the dif- 
ference between two Frank constants and only present 
one stiffness coefficient 71. (This approximation is accu- 
rate only near the Pomeranchuk quantum critical point.) 
More importantly, because n^' b is odd under parity trans- 
formation for I = 1, a new 7 2 term appears, which is 
of cubic order in the order parameter field n M ' b , and it 
is linear in derivatives. This term is allowed by all the 
symmetry requirements, including time reversal, parity, 
and rotation symmetries. This term has no important 
effects in the disordered phase and in the a-phase, but 
it leads to a Lifshitz-likc inhomogeneous ground state 
with spontaneous chirality in the /3-phase in which par- 
ity is spontaneously broken. We will discuss this effect 
in detail in Section [Villi The coefficient of 71,2 will be 
presented in Eq. (|8.19|) . Similarly, for all the odd values 
of I, we can write a real cubic 7 2 term satisfying all the 
symmetry constraints as 



Xb 



(3.6) 



However, this term corresponds to high order corrections, 
and is negligible (irrelevant) for I > 3. 

Similarly to the approximate gauge symmetry for the 
fermions in Eq. (|2.12|) . the 7 2 term can also be repro- 
duced by a non-Abelian gauge potential defined as 



iA x a (x)(T x )^=e Xtlu n Xa (x), 



(3.7) 



where T x = —ie\^ v is the generator of the SU{2) gauge 
group in the vector representation. Then Eq. 



be written as 



F, 



d (n) = -n{(d a 6 lu ,-igA&T x ) lu ,)n vb y 
+ lxg\e Xllv n Xa n vb f 
= ^2 {71(^6 + g n a x ri b ) 2 



jig \n a xiii| t, 



i 2 }- 



(3.8) 



with g = 72/(271). 



B. The 3D systems 

In 3D, the order parameter in the Ff 1 channel Pomer- 
anchuk instabilities can be similarly represented by a 
3 x (21 + 1) matrix. Here we only consider the simplest 
case of the p-wave channel instability (I = 1), which has 
been studied in Ref. 0, H3, EH H2] under different con- 
texts. In the f" channel, the order parameter n^' 1 is a 
3x3 real matrix defined as 



l/fl 



d 3 k 



mkK k b Mk))- (3.9) 



The difference between n^ 1 ' 1 and the triplet p-wave pairing 
order parameter in the 3 He system [l|, |46| is that the 
former is defined in the particle-hole channel, and thus is 
real. By contrast, the latter one is defined in the particle- 
particle channel and is complex. Each column of the 
matrix form n/*' (b = x, y, z) can be viewed as a 3- vectors 
in the spin space as 



ri 2 = n^ 2 , 



n 3 = 



(3.10) 



which represents the spin current in the x, y and z direc- 
tions respectively. The G-L free energy in Ref. [3(| can 
be reorganized as 



F(n) = r'tr(n T n) + (v[ + ^)[tT(n T n)] 2 ~^tr[(n T n) 2 



r'(|ni| 2 + |n 2 | 2 + |n 3 | 2 ) + ^(l"i 



"2 



"3 



2 ) 2 +« 2 {l 



m x n 2 \ 



+ \n 2 x n 3 \ + \n 3 x n x 



(3.11) 



where the coefficients r',v' 12 will be presented in Eq. 
(|4.19j) . Similarly, the a-phase appears at v 2 > which 
favors that ri\ || ri 2 || 713, and leaves their ratios arbi- 
trary. The /3-phase appears at v 2 < which favors that 
vectors rii i2j 3 are perpendicular to each other with equal 
amplitudes |ni| = |n 2 | = \ri 3 \. 

Similarly, we present the gradient terms in the G-L free 
energy as 



(3.12) 
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a-phase 



I = 2 



FIG. 1: The a-phases in the Ff and F 2 a channels. The Fermi 
surfaces exhibit the p and d-wave distortions, respectively. 



where the coefficient -f[ 2 will be presented at Eq. (|8.36[) . 
Again, we neglect the difference among three Frank con- 
stants. The 72 term can be represented as 

7 2 1 (d x n 2 - d y ni) ■ (r?i x n 2 ) 
+ {d y ri 3 - d z n 2 ) ■ (n 2 x n 3 ) 
+ {d z ni - d x n 3 ) ■ (n 3 x ni)|. (3.13) 

It can also be represented in terms of the non-Abelian 
gauge potential as in Eq. 



IV. MEAN FIELD PHASES IN THE F? 
CHANNEL 

In this section, we discuss the solution to the mean field 
Hamiltonian Eq. (|2.10p . for the ordered a and /3-phases 
in both 2D and 3D. 



The value of n can be obtained by solving the self- 
consistent equation in the a-phase 



dk 



l/f(0)| 



(27r)3 -{n/(eT(fe))-»/«i(fc))}cos(W*). 

(4.3) 



where ?i/ (£-[■(&)) and nf(£,i(k)) are the Fermi functions 
for the up and down electrons respectively. The dis- 
tortions of the Fermi surfaces of the spin up and spin 
down bands are given by an angle-dependent part of their 
Fermi wave vectors: 

Skp-t \(6) 2a- 1 o , . a -2a 2 ,. 

— , ix ' = x 2 ±(x-\ x d )cosW 

k F 4 2 

- ax 2 cos 2 W ± (2a 2 - b)x 3 cos 3 W + 0(x 4 ), 

(4.4) 

where we introduced the dimensionless parameter x = 
n/lvFkp), where vp and hp are the Fermi velocity and 
the magnitude of the Fermi wave vector in the FL phase. 
This solution holds for small distortions and it is accurate 
only close to the quantum phase transition. By inspec- 
tion we see that in the a-phases the total spin polariza- 
tion vanishes. More importantly, under a rotation by tt/1, 
the charge and spin components of the order parameter 
both change sign, i.e. a rotation by ir/l is equivalent to 
a reversal of the spin polarization. 

The single particle fermion Green function in the a- 
phase at wave vector fc and Matsubara frequency u) n is 



G(k, iuj. 



2 ^ ir,i„ — ft 



l-cr, 



iuj n -£_i(k) iu> n -£i(k) 



B. The 2D /3-phases 



}. (4.5) 



A. The 2D a-phases 

The a-phase is characterized by an anisotropic distor- 
tion of the Fermi surfaces of up and down spins. In 
this phase s z is as a good quantum number. It is a 
straightforward generalization of the nematic Fermi liq- 
uid for the case of the spin channel Q. For example, 
the Fermi surface structures at I — 1,2 are depicted in 
Fig. Q] The quadrupolar, I = 2, case is the nematic- 
spin-nematic phase jj, H| . Without loss of generality, we 
choose fix = n z, and \n 2 \ = 0. The mean field Hamilto- 
nian H a for the a-phase becomes 

H a ,l = j -^^(k)[e(k)-^-ncos(W k )a z )}^{k). 

(4.1) 

The dispersion relations for the spin up and down elec- 
trons become 

£ u (fc) = e(k)- fiTncos(W k ), (4.2) 



The /3-phase appears for v 2 < 0, which favors n\ _L ri 2 
and \ni\ = \n 2 \. Like the case of ferromagnetism, the 
Fermi surfaces split into two parts with different vol- 
umes, while each one still keeps the round shape undis- 
torted. However, an important difference exists between 
the /3-phase at I > 1 and the ferromagnetic phase. In 
the ferromagnet, the spin is polarized along a fixed uni- 
form direction, which gives rise to a net spin moment. 
On the other hand, in the /?-phase with orbital angu- 
lar momentum I > 1, the spin winds around the Fermi 
surface exhibiting a vortex-like structure in momentum 
space. Consequently in the /3-phase the net spin moment 
is zero, just as it is in the a-phase. (Naturally, partially 
polarized versions of the a and /3-phases are possible but 
will not be discussed here.) In other words, it is a spin 
nematic, high partial wave channel generalization of fer- 
romagnetism. In the previously studied cases of the Ff 
channel in Ref. (30j . it was shown that in this phase 
effective Rashba and Dresselhaus terms are dynamically 
generated in single-particle Hamiltonians. The ground 
state spin configuration exhibits, in momentum space, a 



(a) Rashba (ui = 1) 




(b) Dresselhaus (w 



FIG. 2: The /3-phases in the Ff channel. Spin configurations 
exhibit the vortex structures in the momentum space with 
winding number w = ±1, which correspond to Rashba and 
Dresselhaus SO coupling respectively. 



vortex structure with winding number w = ±1 depicted 
in Fig. H 

Here we generalize the vortex picture in momentum 
space in the f" channel to a general Ff 1 channel. We 
assume |ni| = \n2\ — fi. Without loss of generality, we 
can always perform an 5*0(3) rotation in spin space to 
set n\ |j x, and n-i || y. Then, much as in the B phase of 
3 He, the mean field Hamiltonian Hpj for the /3-phase in 
angular momentum channel I can be expressed through 
a d- vector, defined by 



d(k) = (cos(W k ),sm(W k ),0), 



(4.6) 



as follows 



d 2 k 



V> f (fc) Uk) -fi- nd{B k ) ■ a)] ^(fc), 



(4.7) 



where d(9k) is the spin quantization axis for single parti- 
cle state at k. The saddle point value of n can be obtained 



FIG. 3: The /^-phases in the F£ channel. Spin configurations 
exhibit the vortex structure with winding number w — ±2. 
These two configurations can be transformed to each other by 
performing a rotation around the rc-axis with the angle of tt. 




FIG. 4: The spin configurations on both Fermi surfaces in the 
/3-phase (F ; a channel) map to a large circle on an S 2 sphere 
with the winding number I. 



by solving the self-consistent equation 

2f? ( d 2 k / ^ ^ \ 

WW\ =/^(»/^ fc ))- n /^ fc ))'( 4 - 8 ) 

where the single particle spectra read 

£ u (£) = e(A)-M±n, (4.9) 

which is clearly invariant under spacial rotations. The 
Fermi surface splits into two parts with 

=±x - ^-±( a -b)x 3 + 0(x A ). (4.10) 
kp 2 

The single particle Green function in the /3-phase reads 
lr 1 + a-d 1-a -d 



Gi(k,iu n ) = - j 



- }, (4.H) 
iu n -i]{k) iuj n -i i {k)> 



where a ■ d(k) is the l-th order hclicity operator. Each 
Fermi surface is characterized by the eigenvalues ±1 of 
the helicity operators a ■ d(k). 
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From the mean field theory of a and /3-phases, we can 
calculate the coefficients of the G-L theory, Eq. (|3 .4[) , as 



N{0) 1 



1 



L'2 



2 V |F°| 2 n 
AT(0) j- JV'(O) 2 N"(0) ^ 
32 l L A^(0) J 2N(0)S 
N(0) 



(1 - a - 2a 2 
N"(0) 



36) 



32 Dp k ip 



48 



(-a + 2oT 



■ N(0) 



(4.12) 



where V\ <x do not depend on the value of I at the mean 
field level. N'(Q) and N"(0) are the first and second 
order derivatives of density of states at the Fermi energy 
Ef respectively. They are defined as 



N'(0) 
N"(0) 



— \ e = Ef = (l-2a) 



dN 
~d~c 

d 2 N 



N(0) 
v f k f ' 



6(-a+ 2a 2 - b) 



N(0) 

v 2 k 2 ' 



(4.13) 



Both of them only depend on the non-linear dispersion 
relation up to the cubic order as kept in Eq. 12.51 

It is worth to stress that the coefficients of Eq. (|4.12[) 
were calculated (within this mean field theory) at fixed 
density. Similar coefficients were obtained in the spinless 
system analyzed in Ref. Q at fixed chemical potential. 
The is a subtle difference between these two settings in 
the behavior of the quartic terms. At fixed chemical po- 
tential the sign of b, the coefficient of the cubic term in the 
free fermion dispersion relation, is crucial for the nematic 
phase to be stable. However, as can be seen in Eq. (|4.12[ ). 
the sign of the coefficient of the quartic term v\ is deter- 
mined by several effects: the coefficients a and 6, and that 
of an extra (additive) contribution which originates from 
the curvature of the Fermi surface and hence scales as 
N(Q)/k%. It has been noted in Refs. (H, IH E|] that the 
nematic instability for lattice systems may be a continu- 
ous quantum phase transition or a first order transition, 
in which case it involves a change in the topology of the 
Fermi surface. As shown above, this dichotomy is the re- 
sult of the interplay of the single particle dispersion and 
effects due to the curvature of the Fermi surface. The 
same considerations apply to the coefficients that we will 
present in the following subsection. 

The ground state spin configuration in the /3-phase ex- 
hibits a vortex structure with winding number w — I 
in momentum space. The case of w = 2 is depicted in 
Fig. [3] A. Interestingly, in the case of I = 3, after setting 
d = (cos(36» + 7r/2),sin(36l + 7r/2),0), the effective single 
particle Hamiltonian becomes 

H MF {k) = e(k) + n [- sm{W k )a x + oos(30*)°w] . (4.14) 

This single particle Hamiltonian has the same form as is 
that of the heavy hole band of the 2-dimensional n-doped 
GaAs system [49|, which is results from SO coupling. 



Now we discuss the general configuration of the d- 
vector in the /3-phase in the channel. n\ and ni can be 
any two orthogonal unit vectors on the S 2 sphere. The 
plane spanned by 774,2 intersects the S 2 sphere at any 
large circle as depicted in Fig. [4] which can always be 
obtained by performing a suitable SO(3) rotation from 
the large circle in the xy plane. The spin configuration 
around the Fermi surface maps to this large circle with 
the winding number of I. Furthermore, the configuration 
of winding number ±Z are equivalent to each other up to 
rotation of tt around a diameter of the large circle. For 
example, the case of w = — 2 is depicted in Fig. [3] B, 
which can be obtained from that of w — 2 by performing 
such a rotation around the cc-axis. Similarly, with the 
SOs{3) symmetry in the spin space, the configurations 
with w = ±1 are topologically equivalent to each other. 
However, if the 50g(3) symmetry is reduced to SOs(2) 
because of the existence of an explicit easy plane mag- 
netic anisotropy (which is an effect of SO interactions at 
the single particle level), or by an external magnetic field 
B, then the two configurations with w = ±1 belong to 
two distinct topological sectors. 



C. The 3D instabilities of the p-wave spin triplet 
channel 



The mean field theory for the Pomcranchuk instability 
in the Ff channel has been studied in Ref. [30(. To 
make the paper self-contained, here we summarize the 
main results. 

In the a-phase, taking the special case 77 Ma = nd^zSaz, 
the mean field Hamiltonian reads 



2D, a — 



d 3 k 

(2tt) 3 



tpi(k) e(fe) — ft — T7C7 z cos 6k) ip(k), 

(4.15) 



where 9 is the angle between k and z-axis. The Fermi 
surfaces for the two spin components are distorted in an 
opposite way as 

AfcFu(0) = -(l-a)x 2 ±[x+-a(l-a) 3 x s ]cose 
k F 3 y J 1 3 y ' 1 

- ax 2 cos 2 9 ± (2a 2 - b)x 3 cos 3 6 + 0{x 4 ) 

(4.16) 

In the /3-phase, rotational symmetry is preserved and 
a SO interaction is dynamically generated. With the 
ansatz n^ a = nS^a, the MF Hamiltonian reduces to 

H aD ,f3 = ^(k)(e(k) -11 -fid- fc)^(fc). (4.17) 



The single particle states can be classified according to 
the eigenvalues ±1 of the hclicity operator a ■ k, with 
dispersion relations £ B (^)t,i = e{k) — /i ± 77. The Fermi 



surfaces split into two parts, but still keep the round 
shape for two helicity bands with 



Ak 



Ft, I 



kp 



= ±x - x 2 ± (2a - b)x 3 + 0(x 4 ). (4.18) 



The /3-phase is essentially isotropic. The orbital angular 
momentum L and spin S are no longer separately con- 
served, but the total angular momentum J = L + S = 
remains conserved instead. For the general case of n^ a = 
nD^a, it is equivalent to a redefinition of spin operators 
as S'^ — S v D va 5 a ^, thus Fermi surface distortions remain 

isotropic and J' = L + S' is conserved. 

From the above mean field theory, we can calculate the 
coefficients in Eq. (|3.11[) as 



N(0) 1 



JV(0) f l JV'(Q) 2 
24 l3 [ iV(0) J 



N"{0) i 
5N(0) J 



N{0) (7 -2a -8a 2 + 96), 



180v F k 2 F 



N 2 (Q ) _ N(0) 
90 



45v 2 F k 2 F 



(l-6a + 6a 2 -36), 



(4.19) 



where 
N'(0) 
N"(0) 



dN 
~d^ U 
d 2 N 
de 2 1 



N(0), 



2 -2a 
v f k f 

2(1 - 6a + 6a 2 - 36) 
v f k f 



N(0). 
(4.20) 



Once again, the caveats of the previous subsection on the 
sign of the coefficients of the quartic terms apply here too. 



V. GOLDSTONE MANIFOLDS AND 
TOPOLOGICAL DEFECTS 

In this Section we will discuss the topology of the bro- 
ken symmetry a and /3-phases, and their associated Gold- 
stone manifolds in 2D and 3D. We also discuss and clas- 
sify their topological defects. We should warn the reader 
that the analysis we present here is based only on the 
static properties of the broken symmetry phases and ig- 
nores potentially important physical effects due to the 
fact that in these systems the fermions remain gapless 
(although quite anomalous). In contrast, in anisotropic 
superconductors the fermion spectrum is gapped (up to 
possibly a set of measure zero of nodal points of the Fermi 
surface) . The physics of these effects will not be discussed 
here. 






FIG. 5: The half-quantum vortex with the combined spin- 
orbit distortion for the a-phase at I = 1. The triad denotes 
the direction in the spin space. 



A. Topology of the a-phases 

1. 2D a-phases 

For the 2D a-phases, for which we can set (n fia ) = 
nd^ z 5 a i, the system is invariant under SOs{2) rotation 
in the spin channel, the Z/ rotation with the angle of 2tt/1 
in the orbital channel, and a Z 2 rotation with the angle of 
7T around the x-axis in the spin channel combined with an 
orbital rotation at the angle of it /I. Thus the Goldstone 
manifold is 



[SO L (2) (8 SOs(3)]/[SO s (2) k Z 2 <g> Z,] 
= [SO L (2)/Z,]® [5 2 /Z 2 ], 



(5-1) 



giving rise to three Goldstone modes: one describes the 
oscillation of the Fermi surface, the other two describe 
the spin precession. Their dispersion relation will be cal- 
culated in Section IVII Al 

Due to the Z 2 structure of the combined spin-orbit ro- 
tation, the vortices in the 2D-a-phase can be divided into 
two classes. The first class is the I /I- vortices purely in 
the orbital channel without distortions in the spin chan- 
nel. This class of vortices have the same structure as that 
in the Pomeranchuk instabilities in the density channel 
dubbed the integer quantum vortex. On the other hand, 
another class of vortices as combined spin-orbital defects 
exist. This class of vortices bears a similar structure to 
that of the half-quantum vortex (HQV) in a superfluid 
with internal spin degrees of freedom [5l|, [52|, [53|. An 
example vortex at I = 1 of this class is depicted in Fig. 
[5] where the 7r-disclination in the orbit channel is offset 
by the rotation of tt around the x-axis in the spin chan- 
nel. To describe the vortex configuration for the case of 
the p wave channel, we set up a local reference frame in 
spin space, and assume that the electron spin is either 
parallel or anti-parallel to the z-axis of this frame, at a 
point x in real space. Let (f> = be angular polar coor- 
dinate of x with respect to the core of the vortex. As we 
trace a path in real space around the vortex, the frame in 
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spin space rotates so that the spin flips its direction from 
up to down (or vice versa) as we rotate by an angle of 
7r. (In the d-wave channel the vortex involves a rotation 
by tt/2.) Such behavior is a condensed matter example 
of the Alice-string behavior in the high energy physics 
[54l I55I ]. Another interesting behavior of HQV is that 
a pair of half-quantum vortex and anti-vortex can carry 
spin quantum number. This is an global example of the 
Cheshire charge in the gauge theory [5l|, HU . The elec- 
tron can exchange spin with the Cheshire charged HQV 
pairs when it passes in between the HQV pairs. 

Due to the SO(2)l x SOs(2) symmetry in the Hamil- 
tonian, the fluctuations in the orbital channel are less se- 
vere than those in the spin channel. In the ground state, 
the spin stiffness should be softer than that in orbital 
channel. As a result, an integer- valued vortex should 
be energetically favorable to fractionalize into a pair of 
HQV. However, at finite temperatures, in the absence 
of magnetic anisotropy (an effect that ultimately is due 
to spin orbit effects at the atomic level) the spin chan- 
nel is disordered with exponentially decaying correlation 
functions, and thus without long range order in the spin 
channel. However, the orbital channel still exhibits the 
Kosterlitz-Thouless behavior at 2D where the low energy 
vortex configurations should be of HQV. 

2. 3D a-phases 

Similarly, the Goldstone manifold in the 3D a-phase 
for I = 1 can be written as 

[SO L (3) <g> S0 8 (B)]/[SO L (2) <8> SO s {2) k Z 2 ] 
= {S 2 L ®S 2 S ]/Z 2 , (5.2) 

where again the Z 2 operation is a combined spin-orbit 
rotation at the angle of tt to reverse the spin polariza- 
tion and orbital distortion simultaneously. This Gold- 
stone mode manifold gives rise to two Goldstone modes 
in the density channel responsible for the oscillations of 
the distorted Fermi surfaces, and another two Goldstone 
modes for the spin precessions. The fundamental homo- 
topy group of Eq. (JHHJ) reads m [S 2 L <g> Sf ]/Z 2 = Z 2 . This 
means that the 7r-disclination exists as a stable topologi- 
cal line defect. On the other hand, for the point defect in 
3D space, the second homotopy group of Eq. (|5.2p reads 
K2([Sl ® S|]/Z 2 ) = Z ® Z. This means both the orbit 
and spin channels can exhibit monopole (or hedgehog) 
structures characterized by a pair of winding numbers 
(m,n). As a result of the Z 2 symmetry, (m, n) denotes 
the same monopolcs as that of (—m, —n). 

B. Topology of the /3-phases 

In the 2D /3-phase with I = 1 and w = 1 in the xy- 
plane, the ground state is rotationally invariant, thus 



L z + cr z /2 is still conserved. Generally speaking, the 2D 
/3-phases with the momentum space winding number w 
is invariant under the combined rotation generated by 
L z + wa z /2. The corresponding Goldstone manifold is 

[SO L (2) ® SOs(3)]/SO L+s (2) = SO(3). (5.3) 

Three Goldstone modes exist as relative spin-orbit ro- 
tations around x, y and z-axes on the mean field ground 
state. Similarly, for the 3D /3-phase with 1 = 1, the Gold- 
stone mode manifold is 

[SO L (3) <8> SOs(3)]/SO(3) L+s = 50(3), (5.4) 

with three branches of Goldstone modes. The dispersion 
relation of these Goldstone modes will be calculated in 
Section I Villi The vortex- like point defect in 2D and the 
line defect in 3D are determined by the fundamental ho- 
motopy group tti(SO(3)) = Z2. On the other hand, the 
second homotopy group 7^(50(3)) = 0, thus, as usual, 
no topologically stable point defect exists in 3D. 



VI. THE RPA ANALYSIS IN THE CRITICAL 
REGION AT ZERO TEMPERATURE 

In this section, we study the collective modes in the 
Landau FL phase as the Pomeranchuk QCP is ap- 
proached, > -Ff > -2 in 2D and > F t a > -3 in 3D. 
These collective modes are the high partial wave chan- 
nel counterparts of the paramagnon modes in the 3 He 
system. The picture of collective modes in the p-wave 
channel at 2D is depicted in Fig. [6] 

For this analysis, it is more convenient to em- 
ploy the path integral formalism, and perform the 
Hubbard-Stratonovich transformation to decouple the 4- 
fermion interaction term of the Hamiltonian presented in 
Eq. (|2.4p . After integrating out the fermionic fields, we 
arrive at the effective action 

S cS (n b ) = ~J dr J drdr' (/f)- x (f - 7)n b (r) ■ ft b (?) 
+tr In { + e(V) - n b (r) ■ agi^-iV)}. 

(6.1) 

In the normal FL state, we set n = 0, the fluctuations at 
the quadratic level are given by the effective action 

x n vb (-q,-iiJ n ) (6.2) 

where we have introduced the fluctuation kernel 
L^ a ,vb{q,i^7i) which is given by 
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W») = -{fl)-\l)^u5 ab + (Q^ a (q, lLU n )Q»> b (~q, -w n )) 



FL 



(6.3) 



and 



{Q"' a (q,iuJ n )Q v ' b (-q,-iuj n )) FL = ^nsY. ^{G {FL) {k + q,iio n> +iaj n )a^g a (k)G^ FL Hk,iuj n ,)^g b Ck)}. 



VP j 

k.lLJ 



(6.4) 






A: transverse modes 





B: longitudinal modes 



FIG. 6: The p-wave counterpart of paramagnon modes in 2D. 
Taking into account the spin degrees of freedom, there are 
three (six) transverse triplet modes in 2D (3D), and three 
longitudinal modes respectively. 



is the correlation function of the Q^' a operators, defined 
in Eq. (|2.3[) . in the FL phase (i.e. a fermion bubble). Here 



G^ FL \k, 



1 



iuj n - e(k) 



(6.5) 



is the fermion Green function in the FL phase, and it is 
diagonal in spin space. 

After performing the Matsubara frequency summation, 
we find that the fluctuation kernel in the FL phase is 
given by the expression 



d d k n f (e k _ q/2 ) - n f (e k+q/2 ) 
(2Tr) d uj + ir] + e k _ q/2 - e-k+q/2 ' 



A 



ah i 



(6.6) 



where we have performed an analytic continuation to real 
frequency u>, and A ab is an angular form factor to be 
defined below. 



A. Two Dimensions 

In 2D, without loss of generality we can choose the 
direction of q along the x-axis where the azimuthal angle 
6 = 0. The fluctuation kernels are given by 



-(FL 



L Jb(l^) = SabS^^Kq 2 + 5 



N(0) f 1 d9 



i 2tt s + ir\ — cos ( 



-A 



ab S a b j, 



uj/{v F q), S = N{0) 



1 



Ff (0)| 



> 0, (6.7) 



where, as before, /i, v = X,y,z are the components of 
the spin vector, and a,b = 1,2 are the two orbital com- 
ponents. The diagonal components of the angular form 
factors are A aa = (cos 2 19, sin 2 10). For s < 1, the angular 
integral can be performed to yield 



2tt 



d6 



cos 2 w 



o 2-7T s + ir) - cos 8 \ sin 10 

{1± (s-iVl-s 2 ) 2 '}. (6. 



2iVl - s 

For s <C 1, we can expand the above integral, and find, 
for I odd, 



L %%iH>u) = 5 ab 5^{Kq 2 + 5 + N(0) 
r -Is 

1 Is 2 — is, for a = 2 



Is 2 - il 2 s 3 , for a = 1 



(6.9) 



Here, since we have chosen q along the x axis, the compo- 
nent a = 1 denotes the longitudinal component (parallel 
to the direction of propagation q) and a = 2 is the trans- 
verse component. Similarly, for I even, we get 



L %%fau) = S ab S^{ K q 2 + 6 + N(Q) 

x { ls ]- 2 % 3 f Ta = l. (6.10) 
I — is z — zrs, for a = 2 v 1 



When I is odd, the transverse modes are overdamped 
and the longitudinal modes are underdamped. For ex- 
ample, in the case of I = 1, the dispersion relation at the 
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critical point 8 — > + can be solved as 

,3 



for the transverse mode, and 



. nv F q 



JV(0) 



2N(0) 



v F q 



(6.11) 



(6.12) 



for the longitudinal mode. In contrast, when I is even, 
such as the instability in the F 2 S channel, the transverse 
part is under-damped and the longitudinal part is over- 
damped [H]. The difference is due to the different be- 
havior of the angular form factors for I even and I odd 
respectively. 

In both cases, just as it was found in Ref. [4], the 
dynamic critical exponent is z — 3. By the power count- 
ing, the bare scaling dimension of the quartic terms in 
the GL free energy, with effective coupling constants v\ 
and V2, is (d + z) — 4 where d is the spatial dimension, 
while the scaling dimension of the 72 term linear in spa- 
tial derivatives is (d+ z)/2 — 2. All of these operators are 
irrelevant at zero temperature in 2D and 3D. Thus, the 
critical theory is Gaussian, at least in perturbation the- 
ory. However, it is possible that the above naive scaling 
dimensional analysis may break down at the quantum 
critical point. Various authors have found non- analytic 
corrections to Fermi liquid quantities at the ferromag- 
netic quantum critical point [56j . This may also occurs 
here as well. We will defer a later research for the study 
of these effects. At finite temperatures, the critical region 
turns out to be non-Gaussian [jjj . Both the terms whose 
couplings are 72 and now become relevant. The rel- 
evance of the 72 term does not appear in the usual fer- 
romagnetic phase transitions [57| . and we will also defer 
the discussion to this effect to a future publication. 



where [i, v = x,y,z are once again the three compo- 
nents of the spin vector. For the I — 1 (p-wave) case 
a,b = 1,2,3 are the three orbital components. Using the 
formula 



In 



1 



% S + 17] - 

we arrive at 

- (FL 



hi 



1 



iirQ(s < 1), (6.14) 



L la,Jb(l^) = S ab5^{nq +5 

f iV( )( S 2 -if S ), for a =1,2 
+ l -N(Q)(s 2 + qs 3 ) fora = 3 ' 



.15) 



where only the leading order contribution to the real and 
imaginary parts are kept. The dispersion relation at the 
critical point S — > + can be solved as 



1^1,2(1) 



''~ N(0) 



(6.16) 



for the transverse modes, and 



N(0) 



v F q 



l — 



4 7V(0) 



v F q . 



(6.17) 



for the longitudinal mode. Similarly to the case in 2D, 
the longitudinal channel is weakly damped and other two 
transverse channels are over-damped. Again the dynamic 
critical exponent z = 3, thus naively the critical theory 
is Gaussian at the zero temperature. 



VII. THE GOLDSTONE MODES IN THE 
a-PHASE 



B. Three Dimensions 

In 3D, we can choose the z-axis along the direction of 
q. The diagonal part of the angular form factors is now 
A aa = (2ii^i 5 2ii^i iC o S 2 6»). Assuming that \q\ < k F , the 
fluctuation kernel can be approximated as 



L 



N(0) f 1 



d cos 9 

x s + irj — cos 

1 1 



}• 



s = uj/{v F q), 5 = N(0) 



1^(0)1 3 



>0, 



At the RPA level, the Gaussian fluctuations around 
the mean field saddle point of the a-phase are described 
by an effective action of the form 

= £ 5n " a L »L(<l^n) Sn» b . (7.1) 

The fluctuation kernel in the a-phase is 

x Q"'\-q, -iu n )) a , (7.2) 



.13) where 
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(Q^ a (q,ioj n )Q v ' b (-q,-icj n )) a = ^ J2 ^{G {a) (n, k + q, %u) n , + iiv n )^ 9l<a (k)G^ (n, k, iu n ,)a" g l>b {k)} (7.3) 



is the correlator of the operators Q^ a in the mean field 
theory ground state of the a-phase (again a fermion bub- 
ble). Here G^ a \n,k,iuj n ) is the fermion propagator in 
the a-phase with an expectation value of the (nematic- 
spin-nematic for the I = 2 case) order parameter equal 
to n, 

G (o) (n, k, uu n ) = (itu n - e a (k, n)) * (7.4) 

where 

e a (k, n) = e(k) - (n b ) a ■ a gi,b(k) 

9l,i$) = cosZ0g, 9i,2$) = sin ^fc> ( 7 - 5 ) 

is the fermion dispersion, a matrix in spin space, and 
(fib) a — n z S(,.i is the mean field expectation value of 
the order parameter in the a-phase. 

Since in the a-phase there are spontaneously broken 
continuous symmetries, both in 2D and in 3D, the col- 
lective modes will consist of gapped longitudinal modes, 
i.e. along the direction of the condensate, and gapless, 
Goldstone, modes transverse to the direction of sponta- 
neous symmetry breaking, both in the density and spin 
channels. We will discuss the Goldstone modes in the 
/3-phase in the next section. 

We next comment on the stability of the a-phase in the 
p-wave channel. The GL energies Eq. (|3.5[) and Eq. (|3.12p 
contain a cubic term linear in derivatives. In the ordered 
state, it might induce a linear derivative coupling be- 
tween the massless Goldstone mode at the quadratic level 
through the condensate longitudinal mode, thus leading 
to a Lifshitz instability in the ground state. As we will 
show, this indeed occurs in the p-wave /3-phase. However, 
we will see that in the a-phase the spin (density) channel 
Goldstone modes have the same orbital (spin) indices as 
those of the longitudinal mode, thus they can not be cou- 
pled by Eq. ([33]) and Eq . (|3~T2"]) . Instead, the Goldstone 
modes couple to other gapped modes at the quadratic 
level through linear derivative terms, which do not lead 
to instability at weak coupling, but can renormalize the 
stiffness of the Goldstone modes. 
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FIG. 7: The azimuthal angle <f> for the propagation wavevector 
q for the Goldstone modes in the 2D alpha phase. The spin 
down and up Fermi surfaces can overlap each other by a shift 
at the wavevector K = 2n/vF- 

mode in the density channel, and two branches of Gold- 
stone modes in the spin channel. The fluctuation kernel 
of the a-phases, L^ ivb is a 6 x 6 matrix. Its eigenval- 
ues will thus yield six collective modes of which three are 
the above mentioned Goldstone modes. The other three 
modes are gapped, and are associated with the structure 
of the order parameter in the a-phases. In the low fre- 
quency regime u> <C n, we can neglect the mixing between 
the Goldstone modes and other gapped modes. 

The density channel Goldstone mode is associated with 
the field n z ^, conjugate to the bilinear fermion operator 
Q z ,2, is longitudinal in the spin sector and transverse in 
the charge sector, 

Q 22 (r) = ^(r1K Q/3 HV,)]V^(r), (7.6) 

and describes the Fermi surface oscillation in the 2- 
direction while keeping the spin configuration unchanged. 
On the other hand, the spin channel Goldstone modes 
n sp , x ±iy, conjugate to the fermion bilinears Q x ±iy,i — 
(Qx,i i *Qj/.i)/2, describe spin oscillations while keeping 
the Fermi surface unchanged. 



A. The 2D a-phases 

We consider the 2D a-phases assuming the order pa- 
rameter configuration as (n^b) = n8^ z 5bi- The order 
parameter is thus the operator n Zt \ of Eq. (|2.1[) and 
Eq. (|2.2p . As studied in Sec. IIV Al the Goldstone mode 
manifold S 2 x SOl{2) results in one branch of Goldstone 



1. Density channel Goldstone mode 

The density channel Goldstone field n z _2 behaves sim- 
ilarly to its counterpart in the density channel Pomer- 
anchuk instability [4| . The same approximation as in Ref. 

can be used to deal with the anisotropic Fermi surface, 
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i.e., keeping the anisotropy effect in the static part of the 
correlation function, but ignore it in the dynamic part. 
This approximation is valid at small values of order pa- 
rameter, i.e., x — n/(vFkp) <C 1, where n = (n z ,i). We 
define the propagation wavevector q with the azimuthal 
angle <fi as depicted in Fig. [7] for the I = 1 case. 

The effective fluctuation kernel for the charge channel 
Goldstone mode, which we label by fs, reduces to 



= Kq 2 -N(0) 



is sin 2 (lef)) — Is 2 cos(2Z</>) 

(Z even), 
is cos 2 {l(f>) + Is 2 cos(2l(j)) 
(I odd). 

(7.7) 



where s — ui/vpq. Similarly to the results of in Ref. 
this Goldstone mode corresponding to Fermi surface os- 
cillation is overdamped almost on the entire Fermi surface 
except on a set of directions of measure zero: for I even, 
the charge channel Goldstone mode is underdamped in 
the directions <f> = nn/l, n — 0,1, ... ,1, which are just the 
symmetry axes of the Fermi surface; for I odd, it is under- 
damped instead in the directions (f> = ir(n + 1/2) /I, and 
in this case the Goldstone mode is maximally damped 
along the symmetry axes. 



2. Spin channel Goldstone modes 

On the other hand, the spin channel Goldstone fields 
n x ±i v ,i behave very differently. They only involve "inter- 
band transitions" , leading instead to a fluctuation kernel 
of the form 

1 f d 2 k 

L x+iytl {q,u) = nq 2 + — + 2 J ^—^ CO s 2 (W k ) 



n f [^(k-q/2)]-n f [^(k + q/2)] 
0J + ir, + ^(k- q/2) - £ T (k + q/2) ' 



(7. 



where £,i^(k) — e(k) — ^ =F ncos(Z0fe). They satisfy the 
relation of 



L x -i V s(q,uj) — L x +i Vt i(—q,—uj). 



(7.9) 



A detailed calculation, presented in Appendix [Bj shows 
that for ^, <C 1, the kernel L x ± iy ^ reads 



L x ±iy,i(q, u) = nq 



.„2 



N uj 2 
2\F l a \ti 2 ' 



(7.10) 



which gives rise to a linear and undamped spectrum: 



^±wAQ) = J^^ L fi\q\. (7.11) 



Eq. (|7.1ip has to two important features (due to the 
interband transition): the isotropy of dispersion relation 



at j-, <C 1 in spite of the anisotropic Fermi surfaces, 
and the underdamping of the Goldstone modes. The con- 
tribution to the integral comes from the region around 
Fermi surfaces with the width about 2ncos(Wk)/vF- The 
dependence of the integral on q can be neglected at 
<C 1 because a small q changes the integration area 
weakly, and it thus matters only for high order correc- 
tions in q. The contribution to damping comes from the 
region where two bands become nearly degenerate, i.e., 
cos(Wk) ~ 0. However, the angular form factor also takes 
the form of cos (/#&), which tends to suppress damping. 
As VFq becomes comparable to n, the anisotropy and 
damping effects become more important. 

The linear dispersion relation for the spin-channel 
Goldstone modes at 1 holds regardless of whether 

I is odd or even. This fact is closely related to time re- 
versal (TR) and parity symmetry properties of the order 
parameter n lia . For I odd, n lia is even under TR transfor- 
mation, and hence terms linear in time derivatives cannot 
not appear in the effective action. On the other hand, for 
I even even, although n M<1 is odd under TR, in 2D we can 
still define the combined transformation T' as 



T' = TR(tt/1), 



(7.12) 



under which n pa is even. Here R(n/l) is a real space ro- 
tation by an angle of tt/1. thus, also in this case, terms 
which are linear in time derivative are not allowed in 
the effective action. In contrast, for the case of a ferro- 
magnet at I = 0, TR symmetry is broken, and no other 
symmetry exists to form a combined operation T' that 
will leave the system invariant. As a result, terms lin- 
ear in time derivatives appears in the effective action of 
a ferromagnet. The same arguments apply for phases 
with mixed ferromagnetic and spin nematic order (and its 
generalizations). Furthermore, in the presence of time- 
reversal- violating terms, the two transverse components 
of spin fluctuation become conjugate to each other as in 
the presence of ferromagnetic long range order. In this 
case, only one branch of spin wave Goldstone mode exists 
with a quadratic dispersion relation lufm oc q 2 . 



3. Spin wave spectra 

We assume that the Ftf channel is off-critical, thus in 
the normal state no well-defined spin wave modes ex- 
ist. However, in the a-phase the spin channel Goldstone 
modes carry spin, thus induce a well-defined pole in the 
spin wave spectrum. This can be understood from the 
commutation relation between spin modes and the spin 
channel Goldstone modes 



[S x ± iS y , Q x ^fiy,l] — ^2Q z i. 



(7.13) 



In the a-phase where Q z \ obtains a non-vanish expec- 
tation value, then these two channels become conjugate. 
As a result, the spin- wave gains a sharp resonance and 
should exhibit in the neutron scattering experiment. In 
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contrast, in the normal state, the coupling between this 
two modes is negligible, and thus the resonance disap- 
pears. A similar physics occurs in the SO(5) theory for 
the explanation of 7r-resonance in the underdoped high 
T c cuprates [581 ] - 

The effective coupling constant which mixes the S x + 
iS y and Q x -iy,i operators is a bubble diagram which can 
be calculated as 



n 



(7.14) 



This bubble is dressed by the interaction in the chan- 
nel. The resonant part of the spin correlation function 
(i.e. the contribution of the collective mode pole) be- 
comes 



(S + (q,u>)S-{-q,-uj)) 



7H0) 



2 ui 2 
jFf\W 



iS 



lxg(g»l 2 

■'x+iy,l\Qi 

(7.15) 



For fixed but small q, the spectral function exhibits a 
(5-function peak at the dispersion of the collective mode 

ImxM") = nirv 2 F q 2 n 2 \F?\ 2 5{Lu 2 - cu 2 q ). (7.16) 

which will induce a spin resonance in all directions. It 
is worth to note that in the spin channel the isotropy in 
this dispersion relation at small q persists even deeper in 
the ordered phase. 



B. The 3D a-phase for I = 1 

In the 3D a-phase, we assume the Fermi surface distor- 
tion along the z-axis, and the order parameter configura- 
tion as n^ a — nSp gSag. Similarly to the 2D case, the spin 
up and down Fermi surfaces are related by a overall shift 
at the wavevector K — 2n/vpz. The remaining symme- 
try is SOl(2) <8> SOs(2) which results in four Goldstone 
modes. They can be classified as two density channel 
modes and two spin channel modes. Without loss of gen- 
erality, we choose the propagation wavevector q lies in 
the xz-plane. 

The density channel Goldstone modes describe the 
Fermi surface oscillations in the x and y directions, which 
are associated with the fields n Zi \ and n z ^. By a Leg- 
endre transformation, they are conjugate to the bilinear 
operators Q z ,i and Q z ,2- 



— ,/,t 



W a (r)[<rz, a /3(-iVx)]ipp(r) 



Q*i(f 

Qa(f) = ^«[^, a ?(-iV y )]#(r1 (7.17) 

Following the same procedure of the calculation in 2D, 
we find that the fluctuation kernel of the Goldstone mode 
n z i is 



L z i = nq 2 - ijN(0)s cos 2 6 q + N(0)s 2 cos 29 q . (7.18) 



It is overdamped almost everywhere, except if q lies in 
the equator (9 = ir/2), in which case it is underdamped 
and has a quadratic dispersion. On the other hand, the 
fluctuation kernel of the Goldstone mode of n z 2 reads 



L z . 2 = Kq 2 -iN(Q)s-, 



(7.19) 



which has no dependence on the angle 9 q . Hence, this 
mode is over-damped on the entire Fermi surface. 

The spin channel Goldstone modes of n x ±i Vi 3 in the Ff 
channel behaves similarly to that in the 2D case. We sim- 
ply present their fluctuation kernels at small wavevectors 
as 



J x±iy ,3 



(g>) 



nq 2 



3iV(0) 2 



(7.20) 



where s — lu/vf\K\. The spin wave excitation is also 
dressed by the interaction in the F" channel in the a- 
phase. By a similar calculation to the 2D case, we have 



Xs(q,uj) = (S+{q,uj)S-{-q,-uj)) 

lx°(?»l 2 



J x-\-iy,3 



(<z>) 



(7.21) 



Thus it also develops the same pole as in the spin channel 
Goldstone modes. 



VIII. GOLDSTONE MODES IN THE /3-PHASES 

In this section, we calculate the Goldstone modes and 
spin wave spectra in the /3-phase at 2D and 3D at the 
RPA level. We will show that for I = 1, a Lifshitz-like 
instability arises leading to a spatially inhomogeneous 
ground state. This is because of a dynamically generated 
Dzyaloshinskii-Moriya interaction among the Goldstone 
modes as a result of the spontaneously breaking of parity. 
A G-L analysis is presented to analyze this behavior. 



A. The 2D /3-phases 

Without loss of generality, we consider the /3-phase 
in the Fj 1 channel. We first assume a uniform ground 
state with the configuration of the d-vector d = 
(cos 19, sin 19,0) as defined in Eq. (|4.6p . The correspond- 
ing order parameter i.e., the Higgs mode TXhiggsi is con- 
jugate to operator Ohi ggs as 

O higes (r) = ^|(Qx,i(r) + Q,, 2 (r)). (8.1) 

By performing a relative spin-orbit rotation around the 
z, x, and y-axes on the mean field ansatz, we obtain the 
operators for three branches of Goldstone modes as 



1 



O z (r) = —(Q xfi (f)-Q y , x {r)), 



O x {f) 



L (r). (8.2) 
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I = l,w = 1 



FIG. 8: The three Goldstone modes of O z , O x , , O y , of the 2D 
/3-phase can be viewed as a relative spin-orbit rotation for this 
phase with angular momentum / = 1 and winding number 
w — 1. Here <f) is the azimuthal angle of the propagation 
direction q. 
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2n + VFq 



particle-hole 
continuum 

2n — VFq 



uj = v a3 q 
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FIG. 9: The linear dispersion relation of Goldstone modes of 
the 2D /3-phases (I > 2) at small momentum q. For q > q' , 
the Goldstone modes enter the particle-hole continuum, and 
are damped. 



We define the propagation wavevector of Goldstone 
modes q and its azimuthal angle <p as depicted in Fig. 
[SI For the general direction of q, it is more convenient to 
set up a frame with three axes x' , y' and z, where x' || q 



and y' _L q. We rotate O x and O y into 



o x . 



V 

cos O x — sin 1(f) O y 
sin l<fi O x + cos l<t> O 



(8.3) 



O z , O x ', and O y ' are the generators of a relative spin- 
orbit rotation around the z, x' , and y'-axis respectively. 
Thus, in the following, we call the Goldstone mode of 
O x ' the longitudinal Goldstone mode, and those of O z 
and O y ' are two transverse Goldstone modes. 

The system with d — (cos l<t>, sin l(f>, 0) has the following 



reflection symmetry even in the presence of the q, 



9 k —>2<f>- 9 k , a z — > -o-2, 
<j x — > a x cos 2lcj) + <jy sin 2lcf>, 
a y — > a x sin 21(f) ~ a y cos 21 <p. 



(8.4) 



O z and O y i are even under this transformation while O x > 
is odd. Thus, O x ' decouples from O y > and O z , while 
hybridization occurs between O z and O y > . For a small 
wavevector <C 1 and low frequency | < 1, we can ig- 
nore the mixing between the Goldstone modes and other 
gapped modes. For I > 2, the eigenvalues of the fluctua- 
tion kernel for the Goldstone modes is 



L zz (q,u) « L x , x ,(q,w) w L y > y >(q,cj) 



nq~ 



J 1 N(0) 

m 2 \ F a\ ' 



(8.5) 



where we have neglected the anisotropy among the three 
dispersion relations. A finite hybridization between O z 
and Oy' appears at the order of 0(q l ) 



L zy ,(q,uj) cx iq l 



(8.6) 



which is negligible at small q at I > 2. Thus, the spec- 
trum of the Goldstone modes is linear for I > 2. For 
1 = 2, the hybridization is quadratic in q 



L zy i{q,w) 



■ N(0) 



? 2 (1 



46) < Kq 2 



(8.7) 



and thus must be taken into account. The resulting 
eigenmodes in the transverse channel are O z ±iO y i . How- 
ever, the linear dispersion relation remains at I = 2. 

Similarly to ferromagnets, in there are two Fermi sur- 
faces with unequal volume in the /3-phases. The inter- 
band transition has a gap of 2n and a particle-hole con- 
tinuum of width 2vFq as depicted in Fig. [SJ The Gold- 
stone modes correspond to the interband transition with 
a velocity v gs ~ 2n^/K|F ; a (0)|/A^(0), and no Landau- 
damping effects exist at small q. Naturally, after Gold- 
stone modes enters the particle-hole continuum, at the 
wavevector q' ss 2n/(v gs + vp), the mode is no longer 
long lived and become Landau damped. 

The linear dispersion relation of the Goldstone modes 
holds for all the values at I > 2. This feature is also due 
to the symmetry properties of the n^ a under TR and 
parity transformation. The reasoning here is the same as 
that for the a-phase in the Section IVlI Al 

We next calculate the spin- wave spectra in the /3-phase 
at I > 2. We have the following commutation relations 
as 

[S+e^,e^(O x - iO y )/V2] = iO hig + O z , 
[S z ,O z ]=iO hig . (8.8) 

The effective coupling between S z and O z can be calcu- 
lated as 



o ^ \ -%uN(p) a 



(8.9) 
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FIG. 10: [color online] The dispersion relation of Goldstone 
modes for the /3-phase with I = 1. The longitudinal modes 
O y i has linear dispersion relation, while the two transverse 
modes O z ± iy i are unstable towards the Lifshitz-like instability 
at small momentum q. 
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A B 



FIG. 11: Longitudinal and transverse twists for the Lifshitz- 
like instability in the /3-phase with 1 = 1. A) The longitudinal 
twist with fi2 processing around the e\ axis as described in 
Eq. (|8,20[) . B) The transverse twist with the triad formed by 
fti, n t 2, and hs = hi x fi2 precessing around the e x axis as 
described in Eq. (|8.24p . 



Thus the spin-spin correlation function, dressed by the 
channel interactions, near the resonance of the dis- 
persing transverse collective mode has the form 

lx°l 2 

Xs(<7» = (S z {q,uj)S z (-q, -u)) = - — 

L zz {q,u) 

2N(0) uj 2 
\ F l\ w 2 -^ 2 ^*? 2 ' 

(8.10) 

The spectral function at the resonance reads 

lm Xs (q,oj) = 8n 2 K q 2 5(tu 2 - uj 2 q ). (8.11) 

Similarly, the effective coupling between S+e l ^ q and 
e~ l ^ q (O x — iOy)j\f2 gives the same result, 

Xl ±iy = X° sz (8-12) 
The transverse spin-spin correlation function is 

(S + (q,w)S-(-q,-L0))= j Xs ^ 1 (8.13) 

^x+iy ,x— iy 

which is also dressed by the interactions. 



B. Lifshitz-like instability in the 2D p-wave channel 

For 1 = 1, with the assumption of the uniform ground 
state, the dispersion for the longitudinal mode O z re- 
mains linear as 

-i=4n 2 |Ff|^g 2 . (8.14) 

However, the situation for the transverse modes is dra- 
matically different. The mixing between O z and O y i 



scales linearly with q: 

L zy ,{q,uj) = -L y , z (q,uj) w ^= N (°) x ]^- 

(8.15) 

We diagonalize the matrix and then obtain the eigen- 
modes as O z ±iO y i with the following dispersion relation 

<4, ± = 4^1^(0)1 (^±<*£), (8-16) 

as depicted in Fig. [TU1 where c is a constant at the order 
of 1. 

Clearly, lo\ _ < for small \q\. This means that the 
uniform ground state can not be stable in the F® channel 
due to a Lifshitz-like instability [5^, [6(| . This instabil- 
ity can be understood in terms of the nontrivial effects 
of the gradient term with coefficient 72 in the G-L free 
energy in Eq. (|3.5[) . this term is cubic in the order pa- 
rameter rt M ' b and a linear in spatial derivatives. This 
term leads to an inhomogeneous ground state in the j3- 
phase in which parity is spontaneously broken. A sim- 
ilar phenomenon occurs in the bent-core liquid crystal 
system where a spontaneously chiral inhomogeneous ne- 
matic state arises in an non-chiral system. A similar term 
involving a linear derivative and the cubic order of order 
parameter is constructed in Ref. [61] to account for this 
transition. These inhomogeneous ground states also oc- 
cur in chiral liquid crystal [6J] and helimagnets [43j] such 
as MnSi in which parity is explicitly broken. In contrast, 
such a term is prohibited in the ferromagnetic transition 
[44I |45| , and the nematic-isotropic phase transition [3| in 
the density channel Pomeranchuk instabilities. We will 
see that the G-L analysis below based on the symmetry 
argument agrees with the RPA calculations in Eq. (|8.6[) . 

In order to obtain the values of 71.2, we linearize the 
gradient terms in Eq. (|3.5p in the ground state with d- 
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vector configuration of d = (cos 6, sm9, 0), i.e. 



hi = \J n 2 — 8n\e x + 5hi, 
n 2 = \J n 2 — 8n\e y + Sh 2 , 



.17) 



where Shi _L e x , 8h 2 -L e y and hi _L h 2 is kept. The 
contribution from the Goldstonc modes can be organized 
into 

F grad (0) = n{{d t O x ) 2 + {d l O y f + {d t O z ) 2 } 

+ ^(0 X dyO Z ~ Oyd X Z ) 

+ l2n 2 (d x O x +d y Oy). (8.18) 

With the assumption of the uniform ground state, the 
first 72 term behaves like a Dzyaloshinskii-Moriya term, 
and gives a liner dependence in the dispersion relation as 
appears in Eq. (|8.16[) . By matching the coefficients, we 
arrive at the result 



7i 



72 



V2cN(Q) 
v F k 2 F 



.19) 



The second coefficient 72 in the GL expansion of 
Eq. (|8.18p is a total derivative of the longitudinal Gold- 
stone modes. It does not contribute to the equation of 
motion for the Goldstone modes around the saddle point 
of the uniform mean field ansatz. However, Eq. (|3.5[) ac- 
tually allows a twisted ground ground state in the longi- 
tudinal channel as depicted in Fig. [IT] A. Without loss 
of generality, we assume a pitch vector along the x axis, 
q\\ x, and perform a longitudinal twist of the Goldstonc 
configuration of Hi 2 as 

Hi = h(l, 0, 0), h 2 = n(0, cosgx, — ainqx). (8.20) 

This configuration means that we fix hi, and rotate h 2 
around the x-axis. If the system has a small external 
spin-orbit coupling, it pins the order parameter configu- 
ration. We introduce an effective spin-orbit field h so to 
describe this effect 

V(h so ) = —h so (n Xi i + riyfi). (8.21) 

Then the G-L free energy becomes 

F(n) = 7i« 2 g 2 — J 2 n 3 q — h so h cos qx. (8.22) 

If h so is less than a critical value h SO L defined as 



h so ,L — 



4 7 i : 



(8.23) 



then the Lifshitz instability occurs with the pitch of q c = 
72/1/(271). If h so > h SOi L, the instability due to the 
longitudinal Goldstone mode is suppressed. 

Now let us look at the instability caused by the trans- 
verse Goldstone modes as depicted in Fig. [TT] B. The 



transverse mode O z 
triad of hi, fi2, n.3 : 



»2 



n[ cos e e. 



+ iO y describes the precession of the 
- hi x fi2 as 

sin e cos qx e y — sin e sin qx e z J, 

n{ — sine cosqx e x + (cos 2 sin 2 — cos2gx) e 

2 e i 
sin — sin2gx e z |, 



v 

(8.24) 



where e describes the precession amplitude; hi precesses 
around the x-axis, and hi traces a "figure of eight" orbit 
on the surface of the unit sphere. To linear order of e, 
the free energy cost is 

V(n) = ^{nef + \q l2 (nef + ~{nef. (8.25) 

Hence, when h so is larger than a critical value h so ,T = 
7 2 n 3 /187i < h so x, the instability due to the transverse 
Goldstone modes is suppressed. 

Because h so x > h SOt T, the longitudinal Goldstonc 
channel instability is stronger than that in the transverse 
channel. Thus, in the absence of the external field, the 
ground state exhibits the longitudinal twist of Eq. (|8.20p . 
However, this spiral order can not give static Bragg peaks 
in neutron scattering experiments as occurs in the case of 
a helimagnet, although it can couple to the spin-spin cor- 
relations through a dynamic effect. However, the spatial 
inhomogeneity complicates the calculation. This remains 
an interesting problem which we will not pursue further 
in this work. 

If h so > h SOt L, then both instabilities are suppressed, 
and thus in this regime the ground state is uniform. In 
this case, the Goldstone spectrum has an overall shift 
based on Eq. (|8.18|) as 

wf = ujf + 2\f?\nh so , Lu' t 2 = cj 2 + 2\f?\nh so . (8.26) 

as depicted in Fig. [TU] Then the spectra for the two 
transverse Goldstone modes exhibit a roton like structure 
with a gap of 



A=J2h\f?\(h so -h SOtT ), 



.27) 



at q' c = 7 2 n/(4V2~7i) 



C. Goldstone modes in the 3D /3-phase 

For the 3D /3-phase, we only consider the case of I = 1 
with the ground state configuration of the d-vector as 
d(k) || k. The Goldstone modes behave similarly to the 
2D case. The Legendre conjugation operators of the or- 
der parameter n fJ '' b can be decomposed into the operators 
Oj,j z (J = 0, 1,2; J z — — J, ...J) as eigen-operators of the 
total angular momentum J = L + S in the /3-phase. The 
Higgs mode carries J = defined as 



1 



O hlg (r) = —5^ a {r). 



(8.28) 
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Due to the broken relative spin-orbit symmetry, the rel- 
ative spin-orbit rotations generate the three Goldstonc 
modes O x , O y , and O z 



1 



(8.29) 



which carry total angular momentum J = 1. We choose 
the propagation wavevector q along the z-axis. Due to 
presence of q, only J z is conserved. The relation between 
O x , O y and O z and those in the helical basis 0\ t ±\ and 
Oio is 

lt± i = ^=(O x ±iO y ), O li0 = O z . (8.30) 

In the low frequency and small wavevector regime as 
u, VFq <n< vf^f, we can neglect the mixing between 
Goldstone modes and other massive modes. Similarly to 
the case in 2D, we calculate the fluctuation kernels of the 
Goldstone modes as 



Lio,io(<7, oj) = nq 2 - 



4n 2 |/f| 



r f ,s 2 , ^(0) Q ^ 

L\±i,\±i(q,u) = nq ± 



18 k F 4n 2 |/f| 



.31) 
.32) 



For simplicity, we have neglected the anisotropy in the 
q 2 term in Eq. (|8.32p . Their spectra read 



£ =4n 2 |Ff|( 



+ 3 



2V(0) !8k F 



(j 2 = 0,±l). (8.33) 



Because of the broken parity, the channel of j z = —1 
is unstable, which leads to the Lifshiz-likc instability as 
discussed in the 2D /3-phase. 

Again this Lifshitz instability is due to the nontrivial 
72 term in the G-L free energy of Eq. (|3.12p . To determine 
the coefficients of the gradient terms, we linearize the 72 
term around the saddle point, define the deviation from 
the uniform mean field ansatz as 



n\ = y n 2 — Snfe x + 5n\, H2 — y n 2 — Sn^ey + 5n2 



ri 3 = \Jn 2 - bn\e z + Sn 3 , 



(8.34) 



where 5n\ _L e x , 8ni J- e y , and 5n 3 _L e z . The contribu- 
tion from the Goldstone modes becomes 

FgradiO) = iAid^f + {d l Oyf + {d l O z f} 

- -^neijkOidjOk+^VlfPdiOi. (8.35) 

Similarly to the 2D case, the values of j[ and 7 2 can be 
determined by matching the coefficients of the dispersion 
relation in Eq. (|2.5p as 



(8.36) 



7i ~ «, 7 2 = 75 — rr- 
18v f k f 
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FIG. 12: The Q-phases in a B field (I = 1,2). The larger 
Fermi surface has spin parallel to the B field. The Fermi 
surface distortion is not pinned by the B field. 



where the difference among the three Frank constants is 
neglected. 

Following the same procedure in the 2D case in Section 
IVIII B[ we study the instabilities of the longitudinal and 
transverse twists in 3D /3-phase with an external spin- 
orbit field to pin the order parameter 

V so = -On 8 ' 1 



.37) 



After straightforward calculation, we find that longi- 
tudinal twist occurs at the pitch wavevector qL,3D — 
7172/ (271) with a critical value of field to suppress the 
twist at hgg L = 7|n 3 /(47i), and those of the transverse 
twist are <?t,3D = 9i,3r>/2 and h? s ® T = h so ^/2. Thus the 
conclusion is the same as in the 2D case that the instabil- 
ity of the longitudinal twist is stronger than that of the 
transverse twist. Again, if > h^ L , then the ground 
state is uniform, and the spectrum of the two transverse 
Goldstone modes exhibits a roton-like structure with a 
gap of 



A=J2n\f?\(h ao -h SOtT ), 



(8.38) 



located at qr,3D- 



IX. MAGNETIC FIELD EFFECTS AT ZERO 
TEMPERATURE 



In this section, we discuss the magnetic field effect to 
the order parameter configurations in the a and /3-phases. 
Due to the symmetry constraint, the B field does not cou- 
ple to the order parameter n Ma linearly, and the leading 
order coupling begins at the quadratic level as 

AF(B, n b ) = {u + w)g 2 B 2 tr[n T n]-wg 2 B^B v n tlb n vb 



ug 



'B 2 J2 



2 2 \ ^ 

n b + wg 2_ 

b 



(Bxn b ) 2 , (9.1) 



with g the gyromagnetic ratio. The coefficients u and w 
can be determined from the microscopic calculation by 
using the mean field approach as 

u = 4vx - 2u 2 , w = 4v 2 (at 2D), 

u = Qv\ — 3«2, w = 5v2 (at 3D and I = 1), (9.2) 
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FIG. 13: The spin configuration on the Fermi surfaces in the 
/3-phase (F? channel). With the B field, the large circle splits 
to two small circles perpendicular to the B field with the de- 
velopment of a finite magnetization. The large (small) Fermi 
surface polarizes parallel (anti-parallel) to the B field. 
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FIG. 14: The order parameters in the 3D /3-phase at I = 
1. a) Configuration for B' > B as defined in Eq. (|9.5[) ; b) 
Configuration for B c > B > B' . 



with v li2 denned in Eq. (f4~l"2"l) in 2D and Eq. ([4~1"9"1) in 3D 
respectively. v\ is usually larger than v%, thus it is typi- 
cally positive. This means that the external B field sup- 
presses the magnitude of order parameters as 



n(B)=n(B = 0)Jl- J 



(9.3) 



in both a and /3-phases, where the critical value of B c is 
defined as 



HbB c 



"(0) 

Vpkp kpVF 



< 1. 



(9.4) 



This effect has been studied in Ref. 32J. 

In the a-phase, w is positive. Thus, the spin compo- 
nents of the order parameter of this phase, Hf,, prefers 
to be parallel or anti-parallel to the magnetic field B. 
Consequently, the Fermi surface with its spins polarized 
parallel to the magnetic field becomes larger in size than 
the Fermi surface for the spins pointing in the opposite 
direction. In the case of the nematic-spin-nematic phase, 
the a-phase with angular momentum I = 2, the main ef- 
fect of the magnetic field is to break the symmetry by ir/2 



rotations, i.e. a spacial rotation followed by a spin rever- 
sal, while retaining the symmetry under rotations by w. 
Thus, the magnetic field induces a non-zero component 
of the charge nematic order parameter, the nematic in 
the spin-singlet channel. (Analogs of this effect hold in 
all angular momentum channels). On the other hand, 
for a translation and rotationally invariant system, the 
Fermi surface distortion, i.e. orientation of the order pa- 
rameter in real space, cannot be locked to the direction of 
the B field. As a result, the Goldstone manifold becomes 
\SO s (2) ® SO L (2)]/SO s (2) = SO L {2). 

The B field in the /3-phase also constrains the direction 
of the vectors Hf,. Since w is negative in the /3-phase, 
B prefers to be perpendicular to Hb vectors. Thus in 
the 2D system, Hi, Hi, and B form a triad which can 
be either left-handed or right-handed as depicted in Fig. 
1131 The spin configuration on Fermi surfaces changes 
from a large circle into two smaller circles with a net 
spin polarization along the B field. The triad still has the 
degrees of freedom to rotate around the axis of B, thus 
the Goldstone manifold at 2D is reduced from SOl+s(3) 
to SO L+s (2) x Z 2 . The B field effect in 3D systems is 
more subtle. At small values of B, Hi, Hi and 713 can no 
longer for a triad with B. Instead, Hi, Hi and H3 for a 
distorted triad with the direction of the distortion lying 
in the diagonal direction and parallel to B, as depicted 
in Fig. [TU A. As B grows larger, the three vectors Hi, 
Hi and H3 are pushed towards the plane perpendicular to 
B. For B larger than a value B' , defined as 



B' = B, 



f'2 



v 2 + 4\w\vi/u' 



(9.5) 



Hi, Hi and H3 become co-planar, and with a relative angle 
of 27r/3, i.e. an equilateral triangle, as depicted in Fig. 
[TUB. If B is further increased, Hi, Hi and 713 keep these 
directions but their magnitudes continuously shrink to 
zero at B = B c . 

We next discuss the effect of magnetic field B on the 
Lifshitz-likc instability in the 2D /3-phase at I = 1. We 
assume that B is parallel to the z-direction. As showed 
in Section Will B[ the instability of the longitudinal twist 
described in Eq. (|8.20[) is stronger than that of the trans- 
verse twist. Thus, we will focus here on the case of the 
longitudinal twist. In this case, the order parameter Hi 
precesses around the e\ axis. Thus, B, H\ and Hi cannot 
form a fixed triad uniformly in space. The free energy of 
Eq. (|3.5p now becomes: 

V(n) = -fin 2 q 2 - 7 2 n 3 g - wB 2 n 2 (l + cos 2 qx). (9.6) 

Hence, for B larger than a critical value of B c i, 

B r 



Bi 



y/l + 8wi|w|7i/(m7| 



(9.7) 



the Lifshitz instability of the /3-phasc to a phase with a 
longitudinal twist is suppressed. The effects of an exter- 
nal B field in the 3D /3-phase are more complicated and 
will not be discussed here. 
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Finally, we note that the order parameter n p,a can cou- 
ple to the magnetic field B linearly when other external 
fields are also present. For I = 1, a possible additional 
term in the free energy of the form 



AF(B,j)=j 3 B»j a (r)n»' a , 



(9.8) 



where j a is the a-th component of the electric current. 
To leading order, the mean-field value of the coupling 
constant 73 is 



73 



h 2 e^ B N(0) 
2m*VF vpkp ' 



(9.9) 



where m* is the effective mass and e is the charge of 
electrons. For I = 2, a similar term can be constructed 
as 



AF(B,u) = 7 4 S /i u a (f)n p 



(9.10) 



where u a is the strain field. Due to these terms, in the a- 
phase, the electric current and lattice strain can be used 
to lock the direction of the order parameter in real space 
in the presence of an external magnetic field at I = 1 and 
I = 2 respectively. In the /3-phase, this term will distort 
the round Fermi surfaces into two orthogonal ellipses but 
with different volume. 



along the major and minor axes induces a spin current 
jij.,0, fj owm g m tfi e same (or opposite) direction. But 
for the general direction of J", the induced spin current 
g ows w ith an angle with J". We denote the az- 
imuthal angle between the charge current J" and the 
x-axis as <fi. Then the angle between J" and J^' a reads 
20 or 7r — 2(j) depending on the sign of g. The nature 
of the induced spin current here is different from that 
of the spin-Hall effect in semi-conductors with SO cou- 
pling. In that case, the spin-Hall current always flows 
perpendicular to the electric field, and the spin Hall con- 
ductance is invariant under time-reversal transformation. 
Here, because of the anisotropy of the Fermi surfaces, 
the spin current is perpendicular to the charge current, 
only if the charge current flows along the diagonal direc- 
tion (cj> = ±7r/4, ±37r/4). On the other hand, the d-wave 
phases break time reversal symmetry, thus the induced 
spin current is not dissipationless. 

In the /3-phase, without loss of generality, we can take 
order parameter configuration as in Fig. [3J a, i.e., 



n^ ab = n 



(10.4) 



X. SPIN CURRENT INDUCED BY A CHARGE 
CURRENT IN THE d-WAVE CHANNEL 

In the d-wave (1 = 2) case, the order parameters have 
the structure of the spin-quadruple moments. From the 
symmetry analysis, a spin current J^' a may be induced 
by a charge current J" flowing through the system where 
/i is the spin index, and a, b are the spatial indices. For 
simplicity, we only study the 2D a and /3-phases. In the 
standard quadruple notation, the order parameters n^' 
and n M ' 2 can be represented as n.'*' 1 



n 



f i,xx_ nMV and 



n M:2 _ 2n^ ,xy = 2n fl ' yx . Then we write the formula as 



J^ a = gn^ ab J b , 



(10.1) 



where the matrix n^ ab is related to the order parameter 
n^> a as 



(10.2) 



By the standard the linear response theory, the coefficient 
g can be calculated as 



<J 



(3 - 2o)tt 
ek F v F \F£\ 



(10.3) 



In the a-phase, it is convenient to choose the direc- 
tion of the axes of the reference frame x and y along 
the major and minor axes of the distorted Fermi sur- 
faces, and assume spin quantization along z-axis, so that 
n M ' ab = ne 2 diag{l, — 1}. A charge current J° running 



where e XtV denote the spin direction. We assume that 
the charge current J° flows along the x direction. Then 
two spin currents polarizing along orthogonal directions, 
are induced with the same magnitude. The spin current 
flowing along the x-direction polarizes along e x , while 
that flowing along the y-direction polarizes along e y . If 
we measure the spin current along the spatial direction 
with the azimuthal angle <fi respect to the x-axis, the 
induced spin current along this direction polarizes along 
the direction of cos <j> e x + sin cf> e y . Because in the j3- 
phase, there is an induced SO coupling, spin and orbital 
angular momenta will not be preserved separately. As a 
result, it is impossible in the /3-phase to describe a spin 
current by two separated indices (a spatial index and a 
spin one). The spatial degrees of freedom and the spin 
ones must be mixed together. 

Finally, in a real material, there would always be some 
SO coupling. With even an infinitesimal SO coupling, a 
charge current flowing inside the system can remove the 
degeneracy of the ground states in the ordered phase. In 
other words, in the presence of explicit SO interactions, 
a charge current can pin down the direction of the or- 
der parameter. Therefore, the relative angle between the 
order parameter and the charge current is not arbitrary. 
As a result, to be able to adjust the angle between the 
charge current and the order parameter as we mentioned 
above, some other mechanism is necessary to pin down 
the order parameter such that the order parameter will 
not rotate when we rotate the direction of the charge cur- 
rent. For example, an in-plane magnetic field or a lattice 
potential as background can do the job. 
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XI. POSSIBLE EXPERIMENTAL EVIDENCE 
FOR THESE PHASES 

At present time, we are not aware of any conclusive 
experimental evidence for a spin triplet channel Pomer- 
anchuk instability. However, the a and /3-phases pre- 
sented here are just a natural generalization of ferromag- 
netism to higher partial wave channels. Taking into ac- 
count the existence of the p-wave Cooper pairing phase 
in the 3 He systems[l| and the strong evidence for its ex- 
istence in the ruthenate compound Sr2Ru04 [62| . we be- 
lieve that there is a strong possibility to find these phases 
in the near future. Basically, the driving force for behind 
the Pomeranchuk instabilities in the spin triplet channels 
is still the exchange interaction among electrons, which 
shares the same origin as in ferromagnetism. Although 
the weak coupling analysis we have used here may not 
apply to the materials of interest, as many of them are 
strongly correlated systems, many of the symmetry is- 
sues will be the same as the ones we have discussed here, 
with the exception of the role of lattice effects which we 
have not addressed in detail and which may play a sig- 
nificant role, i.e. by gapping-out many of the Goldstone 
modes associated with the continuous rotational symme- 
try of the models that we have discussed. Nevertheless 
that GL free energies will have much of the same form 
even if the actual coefficients may be different, since we 
typically need a strong enough exchange interaction in a 
non s-wave channel. In the following, we will summarize 
a number of known experimental systems (and numeri- 
cal) which suggest possible directions to search for the a 
and /3-phases. 

a. 3 He : The spin exchange interaction in the Fermi 
liquid state of 3 He is very strong, as exhibited in the 
low frequency paramagnon modes in this system, 
the spin fluctuations are known to mediate the p-wave 
Cooper pairing. The Landau parameter Ff in 3 He 
was determined to be negative from various experiments 
[63l l64l . l65l . |66| , including the normal-state spin diffusion 
constant, spin- wave spectrum, and the temperature de- 
pendence of the specific heat. It varies from around —0.5 
to —1.2 with increasing pressures to the melting point. 
Although F" is not negatively large enough to pass the 
critical point, we expect that reasonably strong fluctua- 
tion effects exist. 

b. URU2SV2 ■ The heavy fermion compound 
URu2Si2 undergoes a phase transition at 17K. The tiny 
antiferromagnetic moment developed in the low tempera- 
ture phase can not explain the large entropy loss. About 
40% density of state density is lost at low temperatures. 
Currently the low temperature phase is believed to be 
characterized by an unknown 'hidden' order parameter 
[67| . An important experimental result of nuclear mag- 
netic resonance (NMR) 68] shows the broadening of the 
line shape below T c . This implies the appearance of a 
random magnetic field in the hidden ordered phase. Re- 
cently, Varma et al. proposed the p-wave a-phase as the 
hidden ordered phase [HI, H3] • They fit reasonably well 



the specific heat jump, and more importantly the jump 
of the non-linear spin susceptibility \3 at the transition. 
The origin of the random field in the NMR experiment is 
explained by the spin moment induced by disorder in the 
p-wave a-phase. However, the a-phase still has Fermi 
surfaces, thus it is difficult to explain the large loss of 
density of states. Further, the a-phase is time reversal 
even, thus its coupling to spin moment must involve B 
field. In the NMR experiment, an external B field is in- 
deed added. It would be interesting to check whether the 
line-shape broadening is correlated with the magnitude 
of B. 

c. Sr3Ru20j : The bilayer ruthenate compound 
Sr3Ru207 develops a metamagnetic transition in an ap- 
plied magnetic field B perpendicular to the c-axis. In 
very pure samples, for B from 7.8 T to 8.1 T, the resis- 
tivity measurements show a strong enhancement below 
1.1 K 14]. Transport measurements in tilted magnetic 
fields, with a finite component of the B field in the ab 
plane, shows evidence for a strong in-plane temperature- 
dependent anisotropy of the resistivity tensor, which is 
suppressed at larger in-plane fields [T^. fl5| . This effect is 
interpreted as a nematic transition for the Fermi surface 
of the majority spin component [14|. This result suggests 
a state which is a superposition of both a charge nematic 
and a nematic-spin-nematic state. On the square lattice, 
the d x 2_ y 2 distortion pattern is more favorable than that 
of d xy . Thus the transition should be Ising-like. In the 
presence of SO coupling, while preserving both parity 
and TR symmetries, the magnetic field can couple to the 
d x 2_ y 2 channel order parameter through terms in the free 
energy of the form 

(B 2 x -B*)B z n Zjl , (11.1) 

which is cubic in B, and 

B-n a n a (11.2) 

(where n a is the charge nematic order parameter), which 
is linear in B. Thus, an in-plane B field can lock the 
orientation of the nematic-spin-nematic order parameter. 
This effect is more pronounce if the system is in a charge 
nematic phase. 

d. 2DEG in large magnetic fields : Currently, the 
strongest experimental evidence for a charge nematic 
(fully polarized) state is in the case of a 2DEG in a 
large perpendicular magnetic fieldjll], G2, G3- In the 
second and higher Landau levels, a huge and strongly 
temperature-dependent resistance anisotropy is seen in 
ultra-high mobility samples, for filling factors near the 
middle of the partially filled Landau level. In this regime, 
the I-V curves are clearly linear at low bias. No evidence 
is seen of a threshold voltage or of broad band noise, 
both of which should be present if the 2DEG would be in 
a stripe state, which is favored by Hartree-Fock calcula- 
tions. Both effects are seen in nearby reentrant integer- 
Hall states. Thus, the simplest interpretation of the ex- 
periments is that the ground state is a polarized charge 
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nematic [9l. [Io|. There is still a poorly understood alter- 
nation effect: the strength of the anisotropic resistance 
appears to alternate between the fully polarized state and 
the state with partial polarization. Although this effect 
could be explained in terms of microscopic calculations 
of the order parameter, it is still possible that the latter 
may suggest some form of partially polarized nematic- 
spin-nematic order, there are no reliable calculations of 
Landau parameters in these compressible phases. 

e. The 2DEG at zero magnetic field : The 2DEG at 
low densities is a strongly coupled system and much work 
has been done on this system in the context of its appar- 
ent metal-insulator transition. What interests us is the 
possibility that this system may have phases of the type 
discussed here. (The possibility of non-uniform "micro- 
emulsion" phases in the 2DEG was proposed recently by 
Jamei and collaborators 69|.) A numerical evaluation to 
the Landau parameter Ff in 2D, performed by Kwon 
et al, ,70] by using variational quantum Monte-Carlo, 
found that Ff is negative and decreasing from —0.19 at 
r s = 1 to —0.27 at r s = 5. On the other hand, Chen et 
al. [7ll l72j investigated the many-body renormalization 
effect to the Rashba SO coupling due to the exchange 
interaction in the F" channel. They found the rcnormal- 
ized SO coupling is amplified significantly at large r s by 
using a local field approximation. More numerical work 
to check whether Pomeranchuk instabilities can occur in 
this system would be desirable. 

/. Ultra-cold atomic gases with ap-wave Feshbach res- 
onance : Another type of strongly interacting system is 
cold atoms with Feshbach resonances. Recently, a inter- 
species p-wave Feshbach resonance has been experimen- 
tally studied by using the two component 6 Li atoms [73[ . 
In the regime of positive scattering length, close to the 
resonance the Landau parameter of Ff should be neg- 
ative and large in magnitude. Thus, this system would 
appear to be a good candidate to observe these phases. 
However, since the p-wave Feshbach resonance is subject 
to a large loss-rate of particles, it is not clear whether 
it would be possible to use this approach to observe a 
stable system with a Pomeranchuk instability near the 
resonance. 

g. How to detect these phases : We also propose sev- 
eral experimental methods to detect the a and /3-phases 
(see also the discussion in Ref.[l^|). For the case of the 
spatially anisotropic a-phases, evidence for strongly tem- 
perature dependent anisotropy in the transport proper- 
ties (as well as the tunability of this effect by either ex- 
ternal in-plane magnetic fields and/or uniaxial stress) as 
seen in Sr3Ru207 and in the 2DEG can provide direct 
evidence for the spacial nematic nature of these phases. 
Spatially nematic phases exhibit anisotropic transport 
properties even in a single-domain sample Tj. More dif- 
ficult is to determine their spin structure. Because no 
magnetic moments appear in both phases, elastic neu- 
tron scattering does not exhibit the regular Bragg peaks. 
Since the Goldstone modes are combined spin and orbital 
excitations, they can not be directly measured through 



neutron scattering. Nevertheless, in the ordered phase, 
as we have discussed spin-spin correlation function cou- 
ples to the Goldstone modes, and develops a character- 
istic resonance structure, which should be accessible to 
inelastic neutron scattering. An experimental detection 
of this resonance and its appearance or disappearance in 
the ordered or disordered phases can justify the existence 
of these phases. On the other hand, Fermi surface config- 
urations and single particle spectra in the a and /3-phases 
are different from the normal state Fermi liquids. If the 
angle resolved photon emission spectroscopy (ARPES) 
experiment can be performed, it can readily tell these 
phases. In the /3-phase, the order parameter is similar 
to the Rashba SO coupling, the method to detect the 
Rashba coupling can be applied here. For example, from 
the beat pattern of the Shubnikov-De Hass oscillations 
of the p(B), we can determine the spin-splitting of two 
hclicity bands. The asymmetry of the confining potential 
certainly will also contribute some part to the final spin- 
orbit coupling. But, when the dynamically generated 
part dominates, it will not sensitive to the asymmetry of 
the confining potential. 



XII. CONCLUSIONS 

In summary, we have studied the Pomeranchuk insta- 
bility involving spin in the high orbital partial wave chan- 
nels. GL free energies are construct to understand the 
ordered phase patterns after the instabilities take place. 
The ordered phases can be classified into a and /3-phases 
as an analogy to the superfluid 3 He A and B phases. 
Both phases are characterized by a certain type of effec- 
tive SO coupling, gives rise a mechanism to generate SO 
couplings in a non-relativistic systems. In the a-phase, 
the Fermi surfaces exhibit an anisotropic distortion, while 
those in the /3-phase still keep the circular or spherical 
shapes undistorted. We further analyze the collective 
modes in the ordered phases at the RPA level. Similarly 
to the Pomeranchuk instability in the spin-singlet density 
channel, the density channel Goldstone modes in the a- 
phase also shows anisotropic overdamping, except along 
some specific symmetry-determined directions. The spin 
channel Goldstone modes are found to exhibit nearly 
isotropic linear dispersion relations at small propagating 
wave vectors. The Goldstone modes in the /3-phase are 
relative spin-orbit rotation modes with linear dispersion 
relation at I > 2. The spin- wave modes in both ordered a 
and /3-phases couples to the Goldstone modes, which thus 
develop characteristic resonance peaks, that can be ob- 
served in inelastic neutron scattering experiments. The 
p-wave channel is special in that the /3-phase can develop 
a spontaneous chiral Lifshitz instability in the originally 
nonchiral systems. The GL analysis was performed to ob- 
tain the twist pattern in the ground state. We also review 
the current experiment status for searching these instabil- 
ities in various systems, including 3 He, the heavy fermion 
compound URu2Si2, the bilayer ruthenate Sr3Ru207, 2D 
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electron gases, and p-wave Feshbach resonances with cold 
fermionic atoms. The S^Rv^Oy seems the most promis- 
ing systems to exhibit such an instability in the 2D d- 
wave channel. However, investigation on the SO cou- 
pling effect is needed to understand the suppression of 
the resistivity anomaly due to the in-plane field. 

There are still many important properties of the spin 
triplet Pomeranchuk instabilities yet to be explored. 
In the paper, we did not discuss the behavior of the 
fermionic degrees of freedom, which are expected to 
be strongly anomalous. Generally speaking, the over- 
damped density channel Goldstone modes in the a-phase 
strongly couple to the fcrmions, which is expected to lead 
to a non- Fermi liquid behavior as in the case of the den- 
sity channel Pomeranchuk instabilities [3,0]. However, 
the Goldstone modes in the /3-phase is not damped as 
I > 2, thus similarly to the case of itinerant ferromag- 
nets, the /3-phase remains a Fermi liquid. The p-wave 
channel in particularly interesting. We have shown in 
Eq. (|2.12[) , that p-wave paramagnon fluctuations couple 
to fermions as an SU(2) gauge field. The linear deriva- 
tive terms in the G-L free energy also become relevant in 
the finite temperature non-Gaussian regime. The Hertz- 
Mills type critical theory for the F" contains new features 
compared to the ferromagnetic ones. We defer to a future 
publication to address the above interesting questions. 
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APPENDIX A: LANDAU INTERACTION 
PARAMETERS 

Landau-Fermi liquid theory is characterized by the in- 
teraction functions which describe the forward scattering 
process between quasi-particles as 

f a p„s&&) = f s {PJ) + f a (f,f)ff a p ■ ^s, (Al) 

where p and p 1 lie close to the Fermi surface. The ex- 
pressions of f s and f a can be obtained through a general 



microscopic two-body SU (2) invariant interaction 

V(f u f 2 ) = V c (n - f 2 ) + V s (n - r 2 )5i • S 2 , (A2) 

where the V c and V s are the spin-independent and de- 
pendent parts, respectively. At the Hartree-Fock level, 
f s ' a {p,p') are 

f'%0) = V c (0)-±V c (p-i?)-^V s (p-i?), 



1 



1 



1. 



fW) = -Mp-p") + -v s (o) + z v s(p-^) 
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(A3) 



f s ' a {p,pi) can be further decomposed into different or- 
bital angular momentum channels as 

f s, a= f f^dcose f^ip-mip-p') in 3D, 
II \ /g/-(p.p')cosZ0 in2D 

where Pi is the /th-order Legendre polynomial. For each 
channel of F^' a , Landau-Pomeranchuk (LP) instability 
Q occurs at 



Fr = N(0)f t ' 



,„/ <-(2Z + l) in 3D, 
<-(2-*,, ) in 2D, 



(A5) 



where iV(0) is the density of states at the Fermi energy. 



APPENDIX B: THE GOLDSTONE MODES OF 
THE SPIN OSCILLATION IN THE ALPHA 
PHASE 

In this section, we calculate the spin channel Goldstone 
modes in the 2D a-phases at small wavevector VFq/n <C 1 
and low frequency uj/n <C 1. The expression for the 
dispersion of L x ±i Vi i(q,uj) is 



1 



L x+i y t i(q,U)) = nq + j— 7 + 2 
n f Mk-l)}-n f [^(k + l)} 

<o + iv + ti(k-i)-h&+iy 



d 2 k 



cos Wk 



(Bl) 



where £f,j.(fc) — e(k) — /z=Fricos 19k- Following the proce- 
dure in Ref. we separate Eq. (IB1[) into a static part 
L x +iy,i{q, w) and a dynamic part M x+ i Vi i(q,u>) as 

L x+i y t i(q,u}) = L x+lyi i(q,0) + M x+lVt i(q,uj). (B2) 

At q = 0,lj = 0, from the self-consistent equation Eq. 
(|4.3|) , the integral cancels the constant term as re- 
quired by Goldstone theorem. The detailed form of the 
static part at small but nonzero q is difficult to evalu- 
ate due to anisotropic Fermi surfaces. Because of the 
breaking of parity in the a-phase, it seems that the lead- 
ing order contribution should be linear to q. However, 
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from the Ginzburg-Landau analysis in Sec. IVH A[ the 
linear derivative term in Eq. (|3.5[) does not contribute 
to the coupling among Goldstone modes, i.e., the uni- 
form ground state is stable in contrast to the case in the 
/3-phase. As a result, the dependence on q should start 
from the quadratic order, bringing a correction to the co- 
efficient k. For simplicity, we neglect this correction for 
it does not cause qualitatively different result. Thus, we 
arrive at 

L x +i Vt i(q,u} = 0) = Kq 2 . (B3) 
The dynamic part, M x +i y> \{q, u>), can be expressed as 
d 2 k cos 2 19 k oj 



M 



n f Mk-l)}-n f [^(k+l)} 



£i(fc-!)-£ T (fc + !) 



(B4) 



To evaluate this integral, we make several simplifications: 
the non-linear part in e(k) is neglected and the linear 
order in q is kept in the denominator. We arrive at 



de. 



2w u> ■ 
d0 k 



cos 2 19k 



irj + 2n cos Wk — qvp cos(9k 
— ui cos 2 Wk 



2n cos 10 k — qvF cos(6>fc 



"t (-/B.(S-|) 



k\ 



2-k uj + it] + 2n cos 19k — qvF cos(6>fc — <f) 2n[2n cos 19k — qvF cos(9k 



(B5) 



where k\ and fea satisfy n>f[£i(ki + q/2)] = and 
Tif[£i(k2 — q/2)] = respectively, and <f> is the azimuthal 
angle of q. k\ and ki can be approximated by 



x 2 ncosWk 
«1,2 = k F (l- — ±{ 

4 VF 



q 



coa(9 k - (f>)) 



+ 0(q 2 ). (B6) 
We now transfer the integral over 9k to an integral 



over z = exp(i#fc) and define the density of the states at 
chemical potential in the ordered state as 



7V<(0) 



k F (l~x 2 /4) 



vftt 



The integral above now reads as: 



(B7) 



iV<(0) f dz z l + z- 



ui 



2iri J z 

iV<(0)cj 

4 ^ 

|z|<l 



IT] 



2n 



Res 



- ( ze -i<t> + z-W) + 0(q 2 ) 

l\2 



{z l + z- l f 



z [2uj + 2irj + 2n(z l + z~ l ) - qv F {ze-^ + z^e^) 



(B8) 



This integral can be calculated by evaluating the residues 
at poles inside the unit circle. There is one pole at 0, 
one pole at oo, and 21 poles, from the solutions of the 



equation: 

2lu + 2ir) + 2n{z l + z~ l ) - qv F {ze~ i<1 ' + z _1 e^) = 0. 

(B9) 

The pole at z = oo is not inside the unit circle and does 
not contribute to the integral. 
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The pole at z = has different behavior for differ- 
ent values of I. The residue is — — ^ vt for I = 1, 

(2n— e l< PqVF) ' 
- 4fi "~8^ V ^ fOT Z = 2 > and _ # fOT a11 1 > 2 - T ° lead " 

ing order, all the residues become —^i at / > 1. Next we 
discuss the poles at the solutions of Eq. (|B9[) . For I > 1, 
not all of these poles are located inside the unit circle. 
However, we will not bother to tell which poles are inside 
the unit circle because we can show that these poles at 



most only gives negligible higher order terms. In the limit 
of small q and u>, Eq. (|B9|) can be solved perturbatively 
as a power series of q and u> as z m = exp(- — 2l ' ) + 
0(q) + 0(uj), where m = 1, 2, . . . , 21. To the leading or- 
der, this type of poles are all simple poles, and the residue 
of 1/ (2uj + 2irj + 2n(z l + z~ l ) - qv F {ze-^ + z^e 1 *)) is 
at the order of O(^). Therefore, the contribution from 
the pole z m is: 



iY <(°^-Res(- 1 



(z l + z~ l ) 



4 ~ x z 2uj + 2irj + 2n(z l + z~ l ) - qv F {ze- i 'l > + z^e^) 
N<{0)u} (2lu + 2ir) - qv F (z m e-^ + z^e 1 ^)) 2 1 



z m (2n) 2 



4// 



(BIO) 



which is negligible to order of 0(w 2 /n 2 ) and 0(q 2 /n 2 ). at I > 1. 

In short, for I > 1, only the pole at z — contributes 
to the integral. The result of the fluctuation kernel is 

L x +i v ,\{q,u) = nq 2 - N <^ lj 2 . (Bll) 

From the self-consistent equation we find that N < (0) = 
yj^t. Therefore, the spin channel Goldstone mode reads 



L x+ iy,i(q,u) = Kq 



2 



2»a|/-( )| 



^-^-r- 2 , (Bi2) 

y 2\F l a \n 2 ' V ' 
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